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1. QMPP: BACKGROUND AND CHALLENGE^ 



Peer production of e-learning content is a growing trend, which will play an 
elementary role in creating, validating, enriching, editing, and updating of 
digital learning content. The concept follows the contemporary trend to 
allow and enable ordinary users to enter the nucleus of digital content 
production. 

Already today a variety of Important factors of success in peer production 
and collaborative work are established and the peer production mechanism 
has been a central element in the development of Linux software, the 
Wikipedia movement and various other shared team work results and 
learner-created reports. Although peer production has been recognized as 
an important factor in e-learning content creation, especially with regards to 
new Web 2.0 tools (such as Wikipedia, Slideshare, Slashdot.org and social 
bookmarking etc.), the conceptualization how to manage and organize peer 
production has not been widely discussed. 

However, the benefits of peer- produced content are obvious. The lead times 
for the provision of learning materials shorten, the costs can be remarkably 
lower and the acceptance of the learners can increase due to the fact that 
peer-produced e-learning content (based on professional experience) can be 
more accurate and attracting than "clinically produced" learning content and 
more coherent regarding the needs and contexts of individual learners or 
specific learner groups. So the key paradigm shift of transforming passive 
receivers of e-learning content into active producers of content within their 
specific areas of knowledge leads to a participation in the creation of 
learning content that also serves as an important motivational factor for 
elaborated learning. Accordingly peer production is not only a novel method 
to produce e- learning content, but also an approach to empower a wide 
variety of professionals to the learning content production. Thus it has also 
an important democratic element in bringing the work-related learning 
content production to the actual level of users, tutors and learning 
supporters. Consequently the Importance of peer production of e-learning 
content is expected to grow especially in the sector of vocational education 
and training as well as In professional continuing education. The problem 
encountered by many organisations however is, that their peer-produced 
learning material is neither created nor assessed in accordance with a 
systematic quality approach. 



Extracted from EFQUEL and http://www.qmpp.net 
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Thus the challenge QMPP Is facing, is harnessing the knowledge, creativity 
and wisdom of learners and other key actors in e-learning by a quality 
approach and various methods of quality assurance. Accordingly the basic 
understanding is that peer production can be assisted and managed by 
exploiting the potential of various forms and types of peer production as 
well as by providing efficient enabling structures and services. 



1.1 The role of work package 3 - introduction and 

OBJECTI VES 

The objective of the Work Package 3 'Benchmarking Peer Production’ is to 
identify key approaches of quality management in peer production of e- 
Learning content by bench- marking peer production practices and processes 
in other areas (such as the creation of technical documentation, joint 
editing efforts etc). The key activities include structured benchmarking of 
other areas of peer production of digital content and the organisation of 
three expert panels. 

In order to reduce redundancy as far as possible. It seemed appropriate to 
unify the originally separated two deliverables of work package 3 - the 
synthesis report on the experts panels and the benchmarking report of peer 
production of digital content - for obvious reasons: the outcomes of the 
expert panels form part of the benchmarking report. 

This report has been authored by Mr. Thomas Fischer and Mr. Thomas 
Kretschmer (Institute for Innovation in Learning) with contributions of the 
following persons: 

♦ Dr. Ulf-Daniel Ehlers (EFQUEL) 

♦ Ms. Kaisa Flonkonen-Ratinen and Mr. Ari-Matti Auvinen (FICI 
Productions Oy) 

♦ Mr. David Riley and Mr. Jose Pinzon (lAVANTE) 

♦ Mr. Thomas Fischer, Mr. Thomas Kretschmer and Dr. Walter 
Kugemann (I LI) 

♦ Ms. Michela Moretti and Ms. Gigliola Paviotti (Scienter Italy) 

♦ Mr. Miguel Angel Munoz Castro (Scienter Spain) 

♦ Ms. Laura Fedeli (University of Macerata). 
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2. METHODOLOGICAL FRAMEWORK 



2.1 Definition of the scope of the project - what is peer 

PRODUCTION? 

Peer production can be defined to inciude the digitai content created, 
edited, enriched by peers, in other words by peopie on the "same 
hierarchicai ievei". The contemporary exampies in the internet of peer 
produced digitai content inciude e.g. YouTube^, Facebook^, biogs, fiickr^, 
siashdot.org^ etc. 

The various dictionaries define the term "peer" as foiiows: 

♦ "a person of equai social standing, rank, age, etc."® 

♦ "a person of the same rank or standing; a iegai equai; a person who is 
equai to another in abilities, qualifications, etc."^ 

Often with the term "peer" is also linked the term "peer group", which is 
defined as follows: 

♦ "a social group composed of people of similar age and status"® 

♦ "a peer group is a group of approximately the same age, social status, 
and interests; generally, people are relatively equal in terms of power 
when they interact with peers"®. 

In the context of involving peers to the educational process, some use also 
the term "peer-to-peer education". Peer-to-peer is often linked in the 
technical sense with the "peer-to-peer networks", which describe mainly the 
technical linking of several computers with another as equals. Some authors 
claim also that the concept of peer-to-peer networks is increasingly evolving 
to an expanded usage as the relational dynamic active in distributed 
networks - not just computer to computer, but human to human. Thus e.g. 
Yochai Benkler claims that associated with peer production are the concepts 
of peer governance and peer property in the digital world. 



^ see http://www. voutube .com 
^ see http://www.facebook.com 
^ see http://www.flickr.com 
^ see http://slashdot.org 

® see http://www.thefreedictionarv.com/peer + (read 16 lulv 2008) 

^ see Webster's Encyclopedic Unabridged Dictionary of the English Language, Random House 1989 
® see http://www.thefreedictionarv.com/peer+qroup (read 16 lulv 2008) 

^ see http://en.wikipedia.org/wiki/Peer group (read 16 lulv 2008) 

of Benkler, see Benkler, Yochai: The Wealth of Networks. Yale University Press, USA 2006. 
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For our purposes it is needed to note that peer-to-peer eLearning often 
refers to such instances, in which the peer learners within an educational or 
training institution are developing eLearning contents to each other for 
limited use^^. However, in this QMPP project our aim is also to utilize the 
peer produced content also outside of one specific educational setting. 

The term of "peer production" in the Internet context has similarities with 
the term "user-created content". User-created content has no widely 
accepted single definition (see e.g. OECD's study on User-created content, 
2007), but according to the guidelines by the OECD in their study, in this 
research paper user-created content is defined with three criteria, which are 
the following: 

♦ content is made publicly available over the Internet 

♦ it reflects a "certain amount of creative effort" 

♦ it is "created outside of professional routines and practices". 

As a conclusion, in our QMPP project, we want to emphasize the 
experiences of peer production, which includes also the strong presence of 
the peer group in the various phases of the learning provision. According to 
our reading of the literature, the user-created content is mainly describing 
the digital artefacts produced by various individuals, as in our approach the 
communicative element is essential - and it takes often place by the strong 
involvement of the peer group. Thus according to the different terms and 
their use we can summarize them in the following picture (see picture 1). 



see e.g. Kotzinos, D. et al.: Online Curriculum on the Semantic Web: The CSD-UoC Portal for Peer- 
to-Peer E-learning. Procee(ding of WWW 2005, May 10-14, 2005, Chiba, Japan. 

OECD - Working Party on Information Technology, 2007: Participative Web: User-created content. 
DSTI/ICCP/IE(2006)7/FINAL. OECD 
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The group emphasis is also highlighted in the discussion of "informal 
learning" - e.g. Jay Cross states that Informal learning Is strongly fueled by 
the communication of peers and that this communication is the critical 
element in informal learning. 



2.2 Quality APPROACHES TO peer production in eLearning 

The quality of eLearning has been discussed and researched in many 
European projects as well as in many international contexts. Ehlers and 
Pawlowski describe, in the discussion on quality of eLearning, one can 
distinguish between three different aspects in the discussion, namely 

♦ different interpretations of quality 

♦ different stakeholders with different perspectives on quality 

♦ different forms of quality (input-quality, process-quality, output- 
quality). 



Cross, J: Informal Learning: Rediscovering the Natural Pathways That Inspire Innovation and 
Performance. Pfeiffer 2006. 
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This discussion has also lead to different interpretations of quality - and 
numerous definitions from various fields are available/'^ As In particular the 
wide discussion on quality in eLearning in general is well captured by the 
recent comprehensive book edited by Ehlers and Pawlowski^^, in this 
context it is not necessary to repeat this discussion, but rather concentrate 
on the key issues which are altering in the peer production of eLearning. 

In the peer production of eLearning the essential feature is that the learners 
are also acting as creators of the content - in the new learning settings the 
separation between an "author" and a "consumer" is blurring. In practice, 
learners are no longer purely consumers but they actively participate in the 
learning process and thus influence it. As the borders between user and 
author are blurring, so do the roles of student and teacher. 

This fundamental feature is also Imposing a different view on quality, as 
quality is often to be defined and assessed by the same group of actors as 
the actual creation of the learning content. However, the quality approach 
to peer production can be more than just an emphasis on self-evaluation 
and its practices. 

Many quality approaches also in eLearning rely on the conventional quality 
cycle. This quality cycle has included - since the writings of W. Edwards 
Deming in the 1950s - the steps of PDCA (Plan, Do, Check, Act). This 
approach has been modified during the last decades in many different ways, 
and also applied into the area of vocational training. Wirth has presented 
that in eLearning the essential steps could be: 

♦ - plan 

♦ - do 

♦ - check 

♦ -compare.^® 

As a hypothesis we can claim, that in the development of a quality approach 
to peer production, the quality approach would mainly address the "process 
quality" issues - in other words: which processes implemented are assisting 
the quality of peer produced eLearning content. In addition, in regarding the 
quality processes, the peer production of the learning materials In their 
quality assessment can also be linked with benchmarking - or even more 
precisely, on "benchlearning". 



Ehlers, U-D. - Pawlowski, J.: Quality in European e-learning: An introduction. In Ehlers, U-D. - 
Pawlowski, J. (eds.): Handbook on Quality and Standardisation in E-Learning. Springer 2006. 
see Ehlers, U-D. - Pawlowski, J. (eds.): Handbook on Quality and Standardisation in E-Learning. 
Springer 2006. 

Wirth, M.A.: An analysis of international quality management approaches in e-learning: Different 
paths, similar pursuits. In Ehlers, U-D. & Pawlowski, J. (eds.): Handbook on Quality and 
Standardisation in E-Learning. Springer 2006. 



Education and Culture 



8 



QAfPP 



WP3: Benchmarking Peer Production 
Report on the Expert Panels 



2.2.1 Quality development of peer production!? 

Ehlers discusses the quality development in what he is referring to as 
"eLearning 2.0”. In describing the phenomenon "eLearning 2.0", he points 
out that it describes a number of developments, trends and points of view, 
which require change from teaching to learning. The new point of view 
essentially connects e-learning with five characteristics: 

1. Learning takes places always and everywhere (ubiquitous) and 
therefore in many different contexts, not only in the classroom. 

2. Learners take on the role of organizers. 

3. Learning is a life-long process, has many episodes and is not (only) 
linked to educational institutions. 

4. Learning takes place in communities of learning (so called communities 
of practice. Learners participate in formal, as well as informal 
communities. 

5. Learning is informal and non-formal, takes place at home, at the work 
place and during leisure time and is no longer centred on teachers or 
institutions. 

In the new environment, learners are highly self-directed, as learning does 
not only take place in institutions, but everywhere, during the course of 
one’s whole life in a number of different episodes, in learning communities 
and social networks, using social software and individually compiled 
contents. Securing and developing quality in such learning scenarios thus 
has to focus mainly on the individual learning processes and the shown 
achievements (performance). The learner's perspective is more important 
than the organizational processes and / or the co called input factors. 
Quality assessment does not take place by using classical methods of 
expert- and standard based quality management, quality assurance or 
control, but by making use of more participative methods and responsive 
designs. The aim of the process is to reach an individualized assessment, 
which relates to the learning process. 

Initially it seems paradox to talk about the quality of eLearning 2.0, as 
quality is often linked with checking by externally imposed standards. 
However, quality can also be understood in a development-oriented way, 
which means the enabling of learners to develop themselves in their own 
learning processes and consequently reach better results as far as quality is 
concerned. In this view, methods of self-evaluation, reflection and peer- 
evaluation are seen as more important. This kind of quality methodology 
does not have anything to do with normative, universally valid, but aims at 
improving the quality of the learning process. 



this chapter is based largely on the writings of Ulf-Daniel Ehlers and his article "Web 2.0 - eLearning 
2.0 Quality 2.0 - Perspectives on a change in learning culture and quality concepts” (manuscript 
2008) 
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In eLearning 2.0 learning approach, the learner has an important role as 
active constructor of learning materials (co-creator), personal learning 
environments and initiator of his or her own learning processes. 
Interestingly, this is a characteristic, which is often felt to be a barrier for 
integrating eLearning 2.0 into formal educational processes. This is because 
the competition of learners and teachers and/or other institutional actors 
during quality assessment seems to be insurmountable and only resolvable 
through a loss of power for the institution. 



2.2.2 Conditions for quaiity of peer production^^ 

As a point of departure, eLearning 2.0 does not require a new mode of 
thinking or method of quality development, such as a new and completely 
altered philosophy of quality - no "educational quality 2.0" is needed. 
However, changed basic conditions and contexts need to be taken into 
account. Doing justice to these different contexts, different questions need 
to be posed when dealing with quality development, different objects 
evaluated, different criteria of quality applied and specific methods of 
quality assurance, enhancement and development used. In short: the role 
of quality development is changing. 

While in traditional learning scenarios it mostly means the checking and 
controlling of quality, in eLearning 2.0 it is becoming more the role of an 
enabler of learning progress. Learning methods and quality development 
are moving closer together. Methods such as feedback, reflection and 
recommendation mechanisms are becoming more important. Typical basic 
conditions, which need to be taken into account in quality development for 
eLearning 2.0 scenarios, are explained in the following: 

♦ From reception to participation: the metaphor used for learning is 
changing. In eLearning 

2.0, quality cannot be tied to the evaluation of a pre- determined 
learning environment or learning contents produced by an expert. Not 
the reception but the active participation is most important, that 
means the question in how far a learning scenario stimulates the 
creation of individual personal learning environments, the compilation 
of individual learning contents and sharing them with others. 



this chapter is based largely on the writings of Ulf-Daniel Ehlers and his article "Web 2.0 - eLearning 
2.0 Quality 2.0 - Perspectives on a change in learning culture and quality concepts” (manuscript 
2008) 
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♦ From inspection to reflection: quality development for eLearning 2.0 
shifts the focus from conformity to a reflection of the learning process. 
Learners are supported in reflecting, recognizing and putting into effect 
their own learning progress, educational strategies, needs, etc. and In 
the course of their actions critically reflect the contribution of 
educational media. The aim is to achieve a personally ideal 
configuration of educational media and strategies, which is 
continuously developed through autonomous reflection. 

♦ From product orientation through process orientation to performance 
and competence orientation: the material that is used for learning and 
the processes of its supplier are not the focus of quality development. 
Quality development focuses on the learners’ performance, their 
individually developed learning products, steps in development and 
similar aspects (for example In e-portfolios), which shape their way to 
decision-making and responsibility. 

♦ From planning education for the leaner to planning education by the 
learner: quality of learning scenarios is often attempted to be achieved 
through careful analysis of the need for education, a comprehensive 
conception phase, feedback as far as the design of learning material 
and development processes are concerned and the evaluation of 
learning processes. In eLearning 2.0, many of these processes shift 
from the supplier of a program to the learner. Quality concepts must 
therefore support the learners in their ability to develop quality 
through reflection, enable learner-oriented forms of evaluation and 
offer the necessary tools for quality development to the learners In 
their PLEs. 

♦ From receiver to developer of learning materials: quality assessment in 
eLearning 2.0 scenarios does not follow the logic of a marketing 
effectiveness research to find out how the materials and characteristics 
of media optimally affect the learning process. It is not about learning 
process taking part in a unified learning scenario. Rather, the focus lies 
on processes of development, flexible usage and the validation of 
social communication processes with other learners. 

♦ From the "learning island" LMS to the Internet as a learning 

environment: eLearning 2.0 approach understand Learning 

Management Systems (LMS) as a mere starting point, as a signpost for 
their own search and use of material from the internet, their 
development and linking to other tools which can be flexibly arranged 
to become personal learning portals. Quality assessment then does not 
focus on materials from the LMS anymore but rather on the learning 
products and perhaps on the learning processes documented in an e- 
portfolio. 

♦ From tests to performance: learning progress and achievements 
become visible not only in tests but rather in the learning process 
documented in portfolios (for example in wikis or web logs), learning 
products and social interactions. 
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2.2.3 Key concepts for the implementation of quality 
in peer production^^ 

Quality assessment of eLearning 2.0 focuses on the learning process. There 
is no use of external standards and inter-individual comparisons (such as 
tests or assessments). Rather, methods of self-evaluation, intra-individual 
development processes are employed for this purpose, which are not made 
via tests but via reflection and evaluation of learning products and e- 
portfolios. Even though eLearning 2.0 is a new development as a trend, 
substantial experiences have already been made with the learning models 
of autonomous learning and learning in communities, which are the basis 
for it, as well as with methods for quality assessment of learning processes. 

In particular, three concepts are worth discussing in more detail, namely 
self-evaluation, e-portfolios, and social recommendation by peers. 

The concept of self-evaluation includes enormous potential for quality 
assessment of learning processes in eLearning 2.0. The aim of it is not a 
complete (summative) assessment of learning achievement, but rather an 
improvement of learning abilities. In scientific literature, positive effects of 
self-evaluating processes on the learning achievements can be found. When 
undertaking these processes, students can gain insights into the profile of 
their own strengths and weaknesses. It has also been shown, that if 
students evaluate their own achievements positively, they aim for more 
challenging objectives, engage in their own learning process more and 
mobilize more personal resources. 

Web-based portfolios (e-portfolios) integrate different media and services. 
Learners collect desired learning products in their e-portfolio, which are 
made in the course of a class or even during the whole course of their 
studies. Students can use electronic portfolio to show competences and 
reflect their learning processes. Learning results, connected with remarks 
by tutors, teachers and peers, feedbacks and personal reflections are 
collected. E-portfolios lend themselves also to quality assessment. Learning 
scenarios supported by e- portfolios emphasize the learning process and 
enable a deeper understanding of learning processes by all participants. 

In eLearning 2.0 communication, feedback and the exchange within a 
learning communities is essential. With the help of social software tools 
collaborations can be conducted and information exchanged, as well as 
evaluated mutually. Three methods are of special significance and first 
experiences have been made: 

♦ social recommendation mechanisms 

♦ peer review method 



this chapter is based largely on the writings of Ulf-Daniel Ehlers and his article "Web 2.0 - eLearning 
2.0 

Quality 2.0 - Perspectives on a change in learning culture and quality concepts” (manuscript 2008) 
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♦ peer assist method. 

Social recommendation mechanisms are defined as those methods that 
serve the purpose of assessing the "true quality" of learning material, in 
contrast to methods focused on experts. According to this method, the 
members of a learning community evaluate materials available online. On 
the one hand this method can be understood as "quality evaluation" in the 
course of which each learning material is assessed by learners. On the other 
hand it is also possible to give learners recommendations - a la Amazon - 
on which learning material is thought to be especially useful, so called social 
recommendations. 

Peer review is a concept that has been introduced a number of times, 
especially in the academic sector. It deals with assessing quality by peers - 
that is colleagues or other learners - giving each other feedback. In the 
area of learning, especially in eLearning 2.0 settings, the peer review can be 
used to attain feedback and quality assurance for results, learning progress 
and aims, which is given from other learners or members of the learning 
community. Peer reflection is a process aimed at creating situations for 
reflecting, in which the peers are asked to encourage the reflection of 
learning processes by means of their own experiences. 

One possibility to check on the quality of learning processes is learning from 
other people's solution, respectively entering a peer learning process with 
others. One model that has recently been gaining more importance is the 
peer assist model. It is a structured reflection in the context of a social 
network, which is carried out via social software. This method is clearly 
distinct from peer review, as its primary aim is to simulate learning 
processes. By employing the method for eLearning 2.0 scenarios, social 
assets are used for further developing one's own solutions or for resolving 
learning difficulties, which come up in the learning process. Structured 
reflection of a learning process is possibly by broaching the issue of the 
learning processes, the results and documented outcomes in the peer assist 
process. 



2.3 The hypothesi s for the QMPP quality model 

This chapter discusses the hypothesis for the QMPP model - in other words 
the project's approach how to achieve and ensure quality in peer produced 
eLearning content. As discussed in the first chapters of this document, the 
endorsed quality approach to peer production is not seen as a standard, but 
rather as a quality process including different stages. There are a number of 
critical aspects to be taken into consideration in these different stages. 
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The quality management challenge in eLearning content produced by peer 
production can, however, undermine the merits of this approach and 
method. The quality work methodology in peer production is at its best 
dispersed and fragmented. Often it has also been claimed, that the very 
nature of peer production is its free fiow and thus any formai mechanism 
(including the quality approach) would be drastically against the creativity 
factor. At the moment there are already a number of useful tools and 
approaches used (such as tools for peer reviews, toois for creating own 
wikis, dictionaries etc.) to ensure and improve the quality of peer produced 
eLearning content. However, it is important to emphasize that peer 
production requires also enabling and supporting structures and their 
effective management. The key issue in this project is to deveiop a holistic 
approach to peer production, which enables also the effective utilization of 
this unique method. 

The importance of peer production of eLearning content will grow especially 
in the sector of vocational education and training as well as in professional 
continuing education. Many organizations face challenges of shortening iife- 
cycle of learning content as well as operational challenges in providing 
required learning content with short lead times and lower costs. However, it 
is also understood that the learning content produced by peers (based on 
professional experience) can be more accurate and attracting than "clinicaily 
produced” learning content by external e-learning experts. 

Peer production has great potentiai in the area of vocationai education and 
training. The future workforce in Europe in many professions has not only to 
access and handle great amounts of information and knowledge, but even 
more importantly to produce various elements of information by themseives 
as an integral part of their work. Peer production is not only a novel method 
to produce eLearning content, but it is also an approach to empower a wide 
variety of professionais to the iearning content production. Thus it has aiso 
an important democratic element in bringing the work-related learning 
content production to the actual level of users, tutors and learning 
supporters. 

The QMPP project aims at contributing to the quality development and 
quality management of peer production of eLearning content. The essential 
work in the project is to deveiop a soiid approach on how to support the 
quality management of peer-produced eLearning content, pilot the approach 
in three different VET entities, and to produce a Joint toolset for the VET 
providers of quaiity management of peer- produced e-learning content. 

The QMPP project itself does not take a position of the eLearning tools used, 
but is developing and implementing a systematic process for the quality 
management of peer-produced e-learning content. 
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2.3.1 Peer production cycle 

Within the QMPP project we have developed a metaphor for the effective 
management of quality in peer production. It includes two important 
elements - these are the "peer production cycle" and the "supporting 
activities". 

The "peer production cycle" includes the following phases: 

♦ benchmarking - identifying of good cases and practices, identifying of 
good digital resources, identifying areas of lacking content, sharing 
learning experiences by sharing learning (b)logs etc. 

♦ creating - (shared) authoring of texts and other resources; creating 
images, audio materials, video materials; creating wikis etc. 

♦ validating - validating content with subject matter experts, validating 
content with peers, rating the validity of the content etc. 

♦ editing - sharing editing responsibilities (from proof-reading to 
translation), undertaking peer reviews, creating alternative 
navigational routes etc. 

♦ enriching - creating additional content materials, publishing individual 
works and team works, sharing or learning (b)logs, adding library 
links, social bookmarking, creating wikis etc. 

♦ updating - monitoring existing content, updating existing content, 

adding specific area content etc. 

However, as stated previously, it is obvious that organizations favouring 
peer production must also have enabling and supporting structures. These 
should include the following: 

♦ enabling policies - organizational opportunities for peer production of 

content (such as time resources allocated for peer production), 

management support for peer production, access to various digital 
resources to be used in content production etc. 

♦ enabling procedures - organizational support for peer production, 

guidelines for peer production and peer reviews, guidelines of 
intellectual property rights, agreement on compensation policies etc. 

♦ enabling processes - practical support of peer production, agreed and 
supported processes and workflows for peer production 

♦ enabling tools - joint and shared tools to be used in peer production to 
provide effective and fluent collaborative work. 
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These essential elements are summarized in picture 3. 

Picture 3 




At this stage it should also be emphasized that in real life these stages are 
not linear or directly sequential. In the creation phase there is naturally 
authoring, re-authoring etc. - thus it can rather be seen that in each phase 
there are subphases. Also it is obvious that, for instance, after the 
validating phase the creation phase is restarted and better content is 
provided for the validation. It is important to note that in the contemporary 
Internet environment these phases can be really fast and that the user 
communities can react really rapidly, if needed. 



2.3.2 Existing toois for peer production 

It should be noted, however, that in many cases there are already existing 
good practices (and also open source tools), which have already been 
tested in various environments. This project is not aiming to develop own 
tools or toolsets. Rather the challenge is to bring together and integrate the 
various tools and approaches into a working model and solution, which can 
easily be utilized within various organizations. 
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The enclosed picture 4 visualizes the various phases of the "peer production 
cycle” and points out some of the already existing practices. 



Picture 4 
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2.3.3 Quality challenges in peer production 

The emphasis of the QMPP project is in testing and validating through the 
pilot the real challenges and opportunities of quality management in peer 
production of eLearning. 

For the design of the experiments in piloting, it is necessary to identify the 
key issues in different phases. The following table (see table 2) summarizes 
the quality management challenges in each phase. 
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Table 1 



Peer 

production 

phase 


Key concerns and key 
questions 


Existing approaches 
and tools 


Benchmarking 


■ What content do we need 
for our learning activities? 

■ Is the required content 
already existing in a usable 
and available form? 

■ Can we access and use the 
content? 

■ Can we edit and enrich the 
content? 

■ Which obligations do we 
have with the existing 
content? 

■ How can we be sure of its 
quality? 

■ Are there (peer) references 
of the content? 


■ Search engines of 
the 1 nternet 

■ Search engines of 
the "hidden web" 

■ Blogs 

■ Social bookmarking 

■ Various user groups 


Creation 


■ What content do we want 
to create? 

■ What kind of a guiding 

structure for the content 

do we provide? 

■ What type of support do 

we provide for the 

creators? 

■ What is the division of 

labour between the 

different actors? 

■ How do we create the 

content? 

■ What routing of the work 
do we use? 

■ What is the timeframe of 
the content creation? 

■ How do we ensure the 
quality of the created 
content? 

■ Who can create content 
(open access vs. qualified 
peers)? 

■ How do we make sure that 
no third party IPRs are 
violated? 

■ How do we ensure the IPR 


■ (Open source) word 

processing and 

office tools 

■ Groupware tools 

■ (Open source) 

visualization tools 

■ Tools for podcasts, 
video casts etc. 

■ Media libraries 

■ Wikis and other 
structured 
environments 
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Peer 

production 

phase 


Key concerns and key 
questions 


Existing approaches 
and tools 




issues? 

■ How do we ensure the 

media richness and 

attractiveness of the 

content? 

■ Which media will we use In 
presentation and how do 
we ensure the required 
media balance? 




Validation 


■ Who should validate the 

content? 

■ Which mechanisms are we 

using in the validation 
(e.g. expert review vs. 

peer review)? 

■ How do we ensure the 

feedback of the validation 
to the creators? 

■ How do we support the 

validation work? 

■ What is the timeframe for 
validation? 


■ Routing tools 

■ Rating tools (e.g. 
giving "stars" to the 
content) 

■ Groupware tools 

■ Direct editing to the 
wikis 


Editing 


■ What kind of editing are 
we promoting? 

■ Who is entitled to edit the 
content (experts vs. all)? 

■ How do we share 

responsibilities of the 
editing work (e.g. 

voluntary dlvislon-of- 

labour vs. free access)? 

■ How do we support the 
validation work? 

■ Is the versioning a part of 

editing (e.g. making 

language versions)? 

■ How is the editing work 
validated? 


■ Dictionaries and 

glossaries (to 

support e.g. proof- 
reading) 

■ Groupware tools 

■ Direct editing to the 
wikis 


Enriching 


■ What kind of enriching are 
we promoting? 

■ How do we enrich the 
existing content? 

■ Who is entitled to enrich 
the content (experts vs. 
all)? 


■ Wikis 

■ Social bookmarking 

and shared 

bookmarks 

■ Digital libraries 

■ 
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Peer 

production 

phase 


Key concerns and key 
questions 


Existing approaches 
and tools 




■ How do we support the 
enriching work? 

■ How is the enriching work 
validated? 

■ Which media can be used 

in enrichment (e.g. 

podcasts, video casts 

etc.)? 




Updating 


■ How do we make sure that 
our content is up-to-date? 

■ How do we organize 
updating? 

■ Who is responsible for 
updating? 

■ What kind of updating are 
we promoting? 

■ Who is entitled to update 
the content (experts vs. 
all)? 

■ How Is updating validated? 


■ Routing tools 

■ Alarms (e.g. based 
on calendars) 

■ Blogs 

■ Groupware 



As one can understand, this table is not complete, but it serves as a 
working metaphor for the pilots to start the planning of their work and their 
key issues. 

However, it should also be emphasized that the "inner circle" Is also 
important in the quality management of peer production. Thus the issues of 
the organizational support are essential In making "quality happen". The 
enclosed table (see table 3) summarizes some of the key issues in the 
enabling and supporting structures of peer production. 



Table 2 



Enabling and 
supporting structures 


Key concerns and key questions 


Enabling policies 


■ How do we organizationally support peer 
production? 

■ How do we allocate time to peer 
production? 

■ How do we provide access to all needed 
resources. Including digital resources? 

■ How do we compensate/award peer 
production? 
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Enabling and 
supporting structures 


Key concerns and key questions 

■ How do we ensure the appropriate 
approach to the 1 PR issues? 


Enabling procedures 


■ How do we organize the support to peer 
production? 

■ What guidelines do we provide for peer 
production in its various phases? 

■ How do we ensure required resources to 
support peer production (e.g. validators of 
content)? 


Enabling processes 


■ How do we communicate of the options of 
peer production? 

■ How do we support the workflows In peer 
production? 


Enabling tools 


■ What tools do we provide for peer 
production? 

■ Which tools do we actively support? 



The challenge of quality management In peer production is Interesting. As 
peer production as a mechanism also to produce eLearning materials and 
content will grow fast, the appropriate quality mechanism can also ensure 
that peer-produced eLearning materials and contents will reach wider 
audiences. 
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3. THE QMPP BENCHMARKING PROCESS 



For the realisation of the expert panels, the project partners elaborated an 
outline with a relatively fixed structure and research questions. This 
procedure was adopted in order to guarantee a certain comparabiiity of the 
results and should help the partners to organise the expert panels. 

The outline comprised of the following parts: 

A) Short I ntroduction to QMPP 

0 Rationale, objectives, envisaged outcomes 

0 Web 2.0 Technologies e.g. blogs, wikis, podcasts, social 

bookmarking, personal learning environments, e-portfolios 
0 Peer production 

0 Quality assessment 

0 Initial and Continuous Vocational Education and Training (I -VET, 
C-VET), Continuous Professional Development (CPD), skills, 

competences 

0 I ntroductory questions: 

a) Is quality in peer production processes with Web 2.0 
applications manageable? 

b) Is quality manageable with traditional approaches (e.g. ISQ, 
EFQM)? 

c) Is there a generic process model of peer production? 

B) Presentation of 3 QMPP Practice Cases 

Short presentation of 3 QMPP practice cases e.g. lAVANTE, 
www.azubi.net . Finish Elevator Company (optional) 

C) Learning Cafe I : Case Assessment & Suggestions for I mprovement 
Three Speed Cafes on the three best ways to develop / to improve the 
quality of peer production in the 3 QMPP practice cases. 

D) Presentation of Changes from Web 1.0 / E-Learning 1.0 ^ Web 2.0 / 
E-Learning 2.0 



E-Learning 1.0 


E-Learning 2.0 


Quality assessed through 

experts 


Quality assessed through learners and 
peers 


Learning platform 


Personal Learning Environment 


Content 


User Created Content 


Curricula 


Learning diaries/ e-portfolios 


Course structure 


Communication 


Tutor availability 


1 nteraction 


Multimedia (Interactivity) 


Social networks / Communities of 
Practice (CoP) 


Acquisition processes 


Participation processes 
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E) Presentation of the Process Model of Peer Production 




♦ Creating 

-(shared) authoring of courses, texts, resources, wikis; creating 
images, audio materials, video materials etc. 

♦ Validating 

-validating content with subject matter experts, validating content 
with peers, rating the content etc. 

♦ Editing 

-shared editing, undertaking peer reviews, creating alternative 
navigational routes etc. 

♦ Enriching 

-creating additional content materials, wikis, pubiishing individual 
works and team works, sharing or learning (b)logs, social bookmarking 
etc. 

♦ Updating 

-monitoring existing content, updating existing content, adding 
specific area content etc. 

♦ Benchmarking 

-identifying of good cases and practices, identifying of good digitai 
resources, sharing learning experiences by sharing learning (b)logs 
etc. 
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F) Presentation of the Peer Production Quality Matrix & reiated Research 
Questions 



O What is the object of the quaiity 
assessment? 

Learning process 

Work/business process 

Degree of 

communication/ participation 

Learning achievement/outcomes 

Learning object, course content, 

resource 

etc. 

0 What are methods and 
instruments to assess/develop 
quaiity? 

Self assessment 
External assessment 
Peer review 
Collaborative dialogue 
etc. 



© What are the dimensions of 

the quality assessment? 

Pedagogical 

Technological 

Economical 

I nstitutional 

Qrganisational 

Cultural 

etc. 



O What are the stakeholders of 

the quality assessment? 

Users 

Experts 

etc. 



G) Learning Cafe II: Benchmarking the Quality Assessment in Peer 
Production Processes with Web 2.0 Applications 

Four Learning Cafes on how to benchmark the quality assessment of 
peer production processes with Web 2.0 applications by relating the 
Peer Production Quaiity Matrix with the Process Model of Peer 
Production 

H) Rapport from the Learning Cafes & Final Plenary Discussion 
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4. SYNTHESIS OF THE EXPERT PANELS 



Structure of this chapter 

This chapter is structured into four sub-chapters: 

1. The first part discloses the structure, setting and setup of the expert 
panels under examination. 

2. The second chapter outlines the applied methodoiogy of anaiysis for 
this synopsis. 

3. The third part presents the results of the panels based on the 
methodology explained within the third chapter. 

4. The fourth chapter provides some conciusions derived from the paneis 
and all previous parts of this paper. 



Purpose 

The objective of this paper is to subsume the results and outline the most 
significant conciusions of the three independentiy conducted expert panels 
on management of peer production of e-learning in three European 
countries i.e. Italy, Spain and Austria. 



Context 

The expert paneis were conducted as part of work package no. 3 
'Benchmarking Peer Production' which is an integral component of the QMPP 
project (Quality Management of Peer Production). It aims to acceierate the 
creation of peer-produced e-learning content by providing a methodology to 
manage its quality.^® 



Extracte(d from http://www.gmpp.net 



Education and Culture 



25 





QAfPP 



WP3: Benchmarking Peer Production 
Report on the Expert Panels 



4.1 The Expert Panels 

The general objective of the expert panels was to explore the experiences in 
peer production mechanism, processes and practices and to collect distinct 
views and applied approaches for peer production of (digital) content for the 
purpose of integrating the results into the design and validation of the QMPP 
quality approach and tools. 

I n this context the focus was on the examination of following questions: 

1. What is the object of the quality assessment in peer production and 
learning 2.0 approaches? 

2. What are the dimensions of the quality assessment in peer production 
and learning 2.0 approaches? 

3. What are methods and instruments to assess/develop quality in peer 
production and learning 2.0 approaches? 

4. What are the stakeholders of the quality assessment in peer 
production and learning 2.0 approaches? 

5. How is the perception of the QMPP Process approach and QMPP Quality 
Matrix in regards to... 

a. Usefulness 

b. Usability 

c. User-friendliness 

d. Comprehensiveness 

e. Comprehensibility 

f. Appropriateness (for different target groups) 

g. Recommendations (for necessary improvements) 

h. Additional comments 

The panels were conducted within the period of June to September 2008. 



As developed within Work Package 2 and 3 

within the Italian expert panel the results were not included into the provided results template but 
based and structured on the basis of a provided questionnaire that included the following questions: 
(1) Definition of key characteristics of a peer, (2) Definition of meaning of "Peer Group/Community", 
(3) Definition of meaning of peer production, (4) Experiences extracted from peer production 
developed within the expert's institution, (5) Products and artefacts created by peers of peer 
production developed within the expert’s institution, (6) Perception of the QMPP Process approach for 
peer production considering (a) the QMPP Peer Production Quality Matrix, (b) the statement that the 
evaluator of the quality of a peer produced output is/should be a co-designer of the output (c) 
strategies and processes for enhancing peer production and (d) the risk of including quality principles 
into a peer production process regarding the potential for creativity and innovation. 
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4.1.1 Participants 

The participants of the panels were professionals to learning content 
production from various organisations from the educational field and 
corporate context such as (for Italy:) Politecnico Milano, Universita Roma 
Tor Vergata, CNR Genova, AmicuccI Formazione, Ufficio StudI Microsoft, 
CIDI Milano, ISFOL; (for Spain:) Universidad de Granada, Fundacion l+D 
del Software Libre, Consorcio Fernando de los Rios, Grupo TADEL / ESEA, 
lAVANTE, Intecna Soluciones, CEVUG; and (for Austria:) AMS 
Oberosterreich, amsbg. Check point eLearning, Dicole Oy, Eotvos Lorand 
University, Free Universtity of Bolzano, Hacettepe University, HCI 
Productions Oy, HTL-TGM Wien 20, ICT&S Center, Intel GmbH, 
Meditrainment, Padagogische Hochschule Rorschach, Primas Consulting, 
Research Studios Austria Forschungsgesellschaft mbH, Scienter Soc., 
Universitat Innsbruck, Universitat Innsbruck, University of Salzburg, and 
University of Duisburg-Essen, lAVANTE, Institute for Innovation in Learning 
[FI M-NewLearnlng]^^ 

Concluding the overall attendance from scientific/educational and corporate 
contexts was relatively balanced. 



4.1.2 Approach 

The expert panels of Spain and Austria were organised as an introductory 
learning cafe and a subsequent face-to-face Meeting/Workshop. The Italian 
panel however was performed via emailed questionnaires and a subsequent 
telephone Interview due to time restrictions of the participants during the 
summer period. For an introductory purpose the (extractions of) previously 
developed documents of the QMPP working packages were made available 
to all participants. Further on a reporting template was provided to support 
the ascertainment and structure of results. 



4.2 Methodology 

4.2.1 Choice of Methodology 

Inevitably an appropriate and reasonable choice of a method of analysis had 
to be made with regards to the design and constitution of the panels and 
their results. Since the panels were built for the purpose of providing 
contextuallzation. Interpretation and understanding of the actors' 
perspectives and given the fact that the examined questions and variables 
are quite complex, strongly interrelated and difficult to measure and the 
results consist of rich, descriptive information rather than statistical data, a 
qualitative analysis approach is an obvious decision. 



Please see the appendix for a complete list of contributors, editors, authors and organisations 
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In accordance with the QMPP Group a qualitative content analysis (focused 
on the summarization) of the result protocols was selected as the 
methodology of analysis. 



4.2.2 Qualitative Content Analysis 

The basic idea of a recapitulatory systematic content analysis is to unify and 
incrementally increase the generality of the source material. 

According to P. Mayring, this approach is predominantly driven by six 
reductive processes: 

1. Elimination 
(Disregarding propositions) 

2. Generalization 

(Substitution of propositions through higher- level propositions) 

3. Construction 

(Construction of one key proposition based multiple single 
propositions) 

4. Integration 

(Ignoring propositions already integrated in a constructed proposition) 

5. Selection 

(Maintaining of selected central propositions) 

6. Grouping 

(Display of several content- wise connected propositions as one) 



According to Mayring (2000, p.l4) 
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For this purpose Mayring offers the following model for the "preparation 
process” of recapitulatory qualitative content analysis of protocol 
documents: 




Figure 1: Model for the "preparation process" of recapitulatory 
qualitative content analysis of protocol documents^ 



On basis of Mayring (2000, p.l4) 
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Further on Mayring provides a plan of procedures for the "process of 
analysis" of qualitative content focused on the summarization of protocol 
documents: 




Figure 2: Plan of procedures for qualitative content analysis focused 
on the summarization of protocol documents^ 



Based on this methodoiogicai structure and guideiines the anaiysis of the 
results will be accomplished by means of the following working steps. 

1. Definition of Problem statement and objects of analysis 

2. Determination of criteria and categories 

3. Examination of source material 

4. I nterpretation and Reporting 



“ On basis of Mayring (2000, p.l9) 
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4.3 Analysis AND Results 

As already briefly described in chapter 4.1 the main objective of analysis is 
the assessment of the following fundamental questions: 

Concepts and Definitions 

5. What are the key characteristics of a peer? 

6. What is the definition/meaning of a "Peer Group/Community"? 

7. What is the definition/meaning of peer production? 

8. What experiences can be extracted from peer production developed 
within the expert's institutions? 

9. Which sorts of products/ artefacts are created by peers within the 
expert's institutions? 

Quality related Outcomes 

1. What is the object of the quaiity assessment in peer production and 
learning 2.0 approaches? 

2. What are the dimensions of the quality assessment in peer production 
and learning 2.0 approaches? 

3. What are methods and instruments to assess/develop quality in peer 
production and learning 2.0 approaches? 

4. Who are the stakehoiders of quality assessment in peer production and 
learning 2.0 approaches? 

5. How is the perception of the QMPP Process Approach and QMPP Quality 
Matrix in regards to... 

a. Usefuiness 

b. Usability 

c. User-friendliness 

d. Comprehensiveness 

e. Comprehensibility 

f. Appropriateness (for different target groups) 

g. Recommendations (for necessary improvements) 

h. Additional comments 

Overarching Areas 

1. How is the appraisal of the statement that the evaluator of the quality 
of a peer produced output is/should be a co-designer of the output? 

2. What are the strategies and processes for enhancing peer production? 

3. How to deal with the risk, that including quaiity principles into peer 
production processes could negatively influence the potential for 
creativity and innovation? 
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4.3.1 Determination of Criteria and Categories 

4. 3. 1.1 Concepts. Definitions and Overarching Areas of Peer 
Production 



Since the concepts and definitions for peer production as well as the 
overarching areas are nearly exclusively based on the source material of the 
Italian expert panel, the results concerning some of these parts will be 
assigned to the questions and definitions under examination without further 
classification. 

Question No. 3 of concepts and definitions for peer production, "What is the 
definition/meaning of Peer Production?" will however be divided into a 
section concerning the "Assessment of the given definition of peer 
production” and another part regarding the "Characteristics of Peer 
Production". 

Question No. 4 of concepts and definitions for peer production, "What 
experiences can be extracted from Peer Production developed within the 
expert’s institutions?" is subdivided into a segment regarding the "contexts 
in which e-learning products are created" and another section concerning 
the "tools used for the production, development and editing". 

Question No. 5 of concepts and definitions for peer production, "Which sorts 
of products/artefacts are created by peers within the expert's institutions?" 
is divided into a part concerning the "final products" and a section regarding 
the "reuse of products". 

The remaining available material of all panels will be assigned to the above 
mentioned questions and subsumed on basis of the six reductive processes 
mentioned within the chapter on qualitative content analysis. 



4. 3. 1.2 Quality related Outcomes 

1. The examination of objects of quality assessment in peer production 

and learning 2.0 approaches is subdivided into the categories of: 

a. Statements concerning the definition of peers, their activities, 
behaviour and the degree of communication/ participation 

b. Statements concerning the organisation and process of learning 

c. Statements concerning the structure and quality of content, 
learning objects, potential achievements/outcomes and the 
challenge of developing a consensus 
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2. The examination of dimension of quaiity assessment in peer production 
and learning 2.0 approaches will be divided into the categories of: 

a. Dimensions of peer production and quality assessment concerning 
institutionai and organisationai issues 

b. Dimensions of peer production and quality assessment regarding 
the economical, cultural and pedagogical aspects 

c. Dimensions of peer production and quaiity assessment concerned 
with content matters 

d. Dimensions of peer production and quality assessment with 
regards to the technological aspects 

3. The examination of methods and instruments of quality assessment in 
peer production and learning 2.0 approaches is subdivided into the 
categories referring to: 

a. Methods and instruments of self assessment 

b. Statements regarding peer reviews and external assessment 

c. Methods and instruments for collaborative dialogue 

4. The examination of stakehoiders of quality assessment in peer 
production and iearning 2.0 approaches wiil not be divided into 
categories. 

5. The examination of questions 5 "How is the perception of the QMPP 
Process Approach and QMPP Quality Matrix" will be subdivided into the 
objects of evaluation. 27 

The quintessence of the experts’ views on the questions regarding the 
concepts and definitions and the quality related outcomes will additionally 
be summarized in simplified illustrations on basis of "cause-and-effect- 
diagrams" that aliow for the identification of flaws, crucial interdependencies 
and the potential for synergetic effects. 

Within the final part of each chapter (concepts and definitions, quality 
reiated outcomes and overarching areas) the essence of ali gathered 
information will then be subsumed by and integrated in contemplation of 
the fundamental questions under examination in order to answer the 
purpose of providing an overview and recapitulatory report. 



27 
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4.3.2 Analysis of the Qualitative Data 

4. 3. 2.1 Concepts and Definitions for Peer Production 



a) What are the key characteristics of a peer? 

Concerning the definition of a peer the expert panels developed the 
following understanding of a peer's characteristics in particular regarding a 
peer's attitude: 

As a part of a community a peer is or should be open-minded, perceptive 
and aware of his competencies and responsibilities in terms of effort, 
attention and reaction rate. In this context peers are expected to possess 
strong relational and auto-critIcal skills, exhibit good relationship 
management and a deep subject matter competence in a specific sector of 
knowledge. 




Figure 3: Key characteristics/ attitudes of a peer 



b) What is the definition/ meaning of a "Peer Group/ Community"? 

A peer group is the group of peers relating to a single person or group. First 
of all it is agreed on the fact that due to the multitude and diversity of peer 
groups and communities it is virtually impossible to determine concrete 
characteristics which hold true for all sorts of peer groups. According to this 
circumstance the following general characteristics of a peer group were 
specified in order to determine an abstract definition. 
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Peer group members are willing to share materials, re-edit existing ones 
and create knowledge and they have a clear and explicit objective to 
support each other in order to grow together. Authority within the group is 
very seldom based on a hierarchy of roles. Rather than that, in most 
communities a levei of members develops naturally and in accordance with 
the degree of participation, quality of contributions and the confidence 
acquired from experience and competencies. Furthermore openness and 
recognizing and exploiting ways of shared communication and operational 
modalities are considered to be essential for a peer group to prosper. This 
becomes even more evident in face of the understanding that sharing 
shouid not be iimited to the exchange of iearning products but also explicitly 
address the sharing of experiences concerning iearning processes, paths 
and projects. 




Figure 4: Definition/ Meaning of a "Peer Group/ Community" 

* As defined in the chapter/ graphic "Key characteristics of a peer" 



c) What is the definition/ meaning of Peer Production? 

Assessment of the given definition of Peer Production 

Given definition of peer production: 

"The digital content for learning created, edited, enriched by peers, in other 
words by people on the same hierarchical level"^® 

The definition is regarded to be adequate for "learners" peer production". 
For "teachers" peer production" however the limitation to digital content is 
considered to be inappropriate and too restrictive. Extending the concept to 
"the structure (project) of educational modules and/or whole courses, or at 
least to the path proposed to the learner" is therefore highly recommended. 
The expression "hierarchical level" however is deemed to induce the 
misleading idea that it only refers to individuals of the same organisation. 



Extracted from the Italian questionnaire 
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Concluding it is perceived to be important to expiicitiy highiight the fact that 
collaboration very often occurs cross-organisational and in the corporate 
sense can even reach across different companies or suppliers. 

In addition another fieid which has been given importance in the expert 
statements is learning by peer- interaction. These learning approaches can 
be summarised by means of the term "Learning 2.0" approach. It describes 
a number of deveiopments, trends and points of view, which require change 
from teaching to learning. The new point of view essentially connects e- 
learning with five characteristics: 

1. Learning takes places always and everywhere (ubiquitous) and 
therefore in many different contexts, not oniy in the classroom. 

2. Learners take on the role of organizers. 

3. Learning is a life-iong process, has many episodes and is not (oniy) 
iinked to educationai institutions. 

4. Learning takes place in communities of learning (so called communities 
of practice: Wenger 1998): Learners participate in formal, as well as 
informal communities. 

5. Learning is informal and non-formal and takes place at home, at the 
work place and during leisure time and is no longer centred on 
teachers or institutions. 

E-learning 2.0 means using social software and learning services, which can 
be combined according to individual needs. The word „can", is very 
significant in this context as technology alone does never determine its use. 



Characteristics of Peer Production 

According to the panels the most significant characteristics of peer 
production are recognizing the vaiue and necessity of compiex processes 
within peer production, such as creation, sharing and editing and the 
readiness to be open and receptive towards inputs from different 
individuais, fields and directions. Besides that it is agreed on the fact that 
sharing the same fieids of interest and objectives is a prerequisite. 
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Figure 5: Definition/ Meaning of Peer Production 

* As defined in the chapter/graphic "Key characteristics of a peer" 
** As defined in the chapter/graphic "Definition/Meaning of a "Peer 

Group/Community" 



In conclusion the understanding of peer production builds on the key 
characteristics/attitudes of a peer and the definition and preconditions of 
peer groups and communities. 




Figure 6: Constitution of the definition of peer production 
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d) What experiences can be extracted from Peer Production 
developed within the expert's institutions? 



Contexts in which e-learning products are created 

The predominant and most common context in which the experts 
experienced the creation of e-learning products are educational activities 
such as teachers' training in educational technology, post graduate courses 
for final projects and collaborative production and sharing of LOs by 
teachers in school projects^®. Besides these, some examples of learning 
content production by students have been experienced, too. In addition to 
that however new and experimental models in ICT use are also tested 
within research activities of institutes. 



Tools used for the production, development and editing 

Since the expert's experiences with most ITD e-learning activities of peers 
concern the use of networked collaborative learning strategies, the 
tendency is to utilise digital material that is not necessarily structured and 
designed according to ODL criteria. Concluding the trend is to collaborate on 
the design and reuse e-learning based processes rather than designing 
actual e- learning contents collaboratively. 

For this purpose learning content management systems such as Moodle and 
collaborative platforms such as C.S.C.W are deployed as supporting tools 
and technologies. Further on web 2.0 applications such as Delicious, Flickr 
and Youtube, as well as wikis, weblogs and traditional ways of online 
collaboration such as e-mail/mailing lists and web forums are the most 
prevailing measures that are yet exploited. Thereunto editors to create 
Learning Objects such as eXe and Reload and content-enhancing and 
exercise supporting tools such as HotPotatoes or CMAP for visualising logical 
maps are known and utilised by the experts, too. Beyond that, classic tools 
to create and update online content like HTML editors or GoLive and 
DreamWeaver are made use of in the same manner. 

In the case of Microsoft Italy, Learning Essentials 2.0 for Microsoft Office 
and the THESIS e-learning product suite are deployed increasingly for the 
production of multimedia contents. 
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Figure 7: Experiences extracted from Peer Production 



e) Which products/ artefacts are created by peers within the 
expert's institutions? 



Final products 

According to the expert's experience the finai products of peer production 
are iearning objects (LOs), storyboards of LOs, articies, reports, course 
notes, book chapters and other sort of documents. Besides, oniine artefacts 
such as wiki pages and tutoriais in Fiash or MS PowerPoint ioom iarge as 
weii. Further on it is observed that finai products such as LOs and tutoriais 
are predominateiy deaiing with subjects whose content is about fieids of 
knowledge that change rather quickiy such as topics related to computer 
and information technologies. 



Reuse of products 

As far as the reuse of products of peer production is concerned the experts 
agree that the smaller the learning units the higher the potential for future 
reuse. Furthermore not only content predominantly produced and reused by 
students such as course notes can be exchanged via shared repositories and 
"recycled" and enhanced by teachers to become complementary educational 
material, but also models like course structures have great prospects for 
reutilisation. 



Education and Culture 



39 



QAfPP 



WP3: Benchmarking Peer Production 
Report on the Expert Panels 




Figure 8: Products/ Artefacts are created by peers of peer 

production 



Summary of key points of concepts and definitions for peer 
production 

In order to cope with the entailed requirements of peer production and 
learning 2.0 environments, in terms of intensified ways and methods of 
communication, interaction, participation, and (self-) assessment, learners 
are required to act autonomously, self- directed, and open-minded, while 
empowering and enabling themselves to enhance and reflect on their own 
competencies, auto-critical skills, and willingness/readiness to share, 
contribute and collaborate. In this regard communities of peer production 
are urged to prepare, set up and adjust their members and the given social, 
environmental and technological surroundings to the shifting requirements 
concerning the understanding of shared needs, interests, objectives and 
products. In accordance with the so far gathered experience, the focus 
needs to be on creating tools, guidelines and policies that enable and drive 
peers to produce preferably small, structured and standardized learning 
objects that are easy to asses and cope with for potential future reuse, and 
transparent in terms of their process of creation. 
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4. 3. 2. 2 Quality related Outcomes 

A) What is the object of the quality assessment in peer production 
and learning 2.0 approaches? 

a. Statements concerning peers, their activities, their behaviour and the 
degree of communication/participation 

In general the behaviours of peers, such as students and potential students, 
the community and other producers of the learning material are considered 
to be more important than knowledge reproduction. In particular the 
process of communication and social interaction, such as those within social 
networks are regarded as crucial objects to the process of quality 
assessment. Therefore the organisation of communication processes 
demands a high attention. 

Statements concerning the organisation and process of learning 
The process of learning, communication and interaction is generally 
regarded to be more significant than reaching pre-defined outcomes of 
learning. 

Regarding the process of learning, the planning and control of activities as 
well as the consideration of evaluation against coordination and 
systematization of spontaneous knowledge building by providing an 
adequate methodology for assessment Is considered to be of crucial 
importance. Furthermore the process of consensus building in respect of 
awareness, trust, cohesion, participation and criteria to achieve a consensus 
is deemed to be of very high significance. 



b. Statements concerning the structure and quality of learning objects, 
content, potential achievements/outcomes and consensus building 

As far as the results and outcomes of quality assessment are concerned, the 
standardization and typology of content entitles and possibilities to record 
and tag these is regarded to be central in terms of effectively organising, 
measuring and filtering knowledge. Besides, the desire to adequately assess 
the achievement of the proposed objectives, the usefulness and suitability 
of content and the level of user satisfaction highly demand for the utilisation 
and development of measuring instruments. 
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Figure 9: Object of quality assessment in peer production and 

learning 2.0 approaches 



B) What are the dimensions of the quality assessment in peer 
production and learning 2.0 approaches? 

a. Dimensions of peer production and quaiity assessment concerning 
institutional and organisational issues 

Regarding the dimensions of peer production, the peer group's respectively 
the final user's adaptation to, and satisfaction with the process and results 
is considered to be pivotal. Facing this chailenge it is essential to heighten 
the ievei of motivation and recognition within the group and every individual 
in it. This includes enabling groups of peer production to foster the personal 
growth, development and social integration of its individuals and their 
learning capabiiities. Therefore this dimension of peer production relies, to 
great extends, on the peer groups' work ethic in regards to expioiting, 
assessing and expanding the social abilities and potentials of each individual 
within the group. 



b. Dimensions of peer production and quality assessment regarding the 
economical, cultural and pedagogical aspects 

Since the social networks are a continuously growing factor for learning and 
the production and obtainment of knowledge the social system surrounding 
a group and its peers is regarded as a key dimension of peer production. 
This leads to the conclusion that the characteristics of peers, the actual area 
of learning (vocational/ professional/ educational) and the specifications of 
the environment (open/close), on which it is based upon, have to be taken 
into account and clearly defined. 
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c. Dimensions of peer production and quality assessment concerned with 
content matters 

As far as the content of peer production is concerned it is regarded to be of 
crucial importance that the way the content was created or composed is 
transparent to the peers within the group. Further on the "up-to-dateness” 
of content in terms of adequate updating the material within an appropriate 
refresh period needs to be ensured in order to provide satisfactory resuits 
for the group and its users. Likewise the rigor and veracity of content has to 
be assured at all times. 



d. Dimensions of peer production and quality assessment with regards to 
the technologicai aspects 

Considering the dimensions of peer production the correct and appropriate 
deployment and combination of tools and instruments is a crucial factor 
within learning 2.0 approaches. In this regard it is pointed out by the 
panels' experts that the peers’ expertise and competence to empioy ICTs 
might often be opposed to their strength concerning the field of knowledge 
in demand. Further on the above mentioned impact of social networks and 
the need to exploit these demands peers to be even more capable to cope 
with ICT. 




Figure 10: Dimensions of the quality assessment in peer production 

and learning 2.0 approaches 
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C) What are methods and instruments to assess/ develop quality in 
peer production and learning 2.0 approaches? 

a. Methods and instruments of self assessment 

In general it is agreed on the fact that quality development in web 2.0 
scenarios demands the empowerment of iearners to assess and evaluate 
their own learning and production. To support this assessment organisations 
are urged to provide a system and structure of indicators based on tools, 
guidelines and decision factors in order to help learners to develop a 
competence in evaiuating their own and others’ progress and results. 



b. Statements regarding peer reviews, externai assessment and 
coliaborative diaiogue 

The need to support iearners to assess the quality of iearning results by 
means of software, guidelines and poiicies of course hoids true for the 
mutual evaluation of learning outcomes, too. But even though it is agreed 
on the circumstance that in web 2.0 scenarios "crowds” often hold more 
wisdom than a singie expert, and that this wisdom needs to be exploited, 
the deveiopment of a consensus is considered to be the cruciai asset. 
Therefore methods such as polls, surveys and questionnaires are regarded 
to be essential. As far as the choice of peers for evaluation and examination 
of materiai produced by other peers is concerned, spontaneousness, 
voluntary or random selection as well as the selection of reference groups is 
regarded to be feasible under certain circumstances. 

Facing the fact that stakehoiders not only develop but also assess their own 
material, nevertheless leads to deliberating a periodical evaluation by 
(external) experts in order to ensure and improve the conclusiveness and 
correctness of outcomes and processes without faliing into traps of common 
knowledge effects, hidden profiies, group polarisation or other phenomena 
that have high potential for amplifying errors due to failures of group 
judgement, information aggregation or consensus buiiding. 



c. Challenges and Problems concerning the assessment of quality 

The key challenge concerning evaluating learning processes and results 
within web 2.0 scenarios cannot be assessed from pre-defined standards 
and measures. This leads to measurements of quality assessment that are 
understandably not always compatible with institutionai guideiines and 
regulations. Since the quality of peer production naturally depends on the 
competence and qualification of its peers. Accordingly the assessment of the 
produced quality, not performed by experts but by the peers themseives, is 
always afflicted to potentially generate so called "garbage-in - garbage-out” 
problems. 
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Figure 11: Methods and instruments to assess/ develop quality in 
peer production and learning 2.0 approaches 



D) Who are the stakeholders of the quality assessment in peer 
production and learning 2.0 approaches? 

Most experts consider the stakeholders of peer production to be "everybody 
who is participating in the process of learning, evaluation and content and 
knowledge creation”. This includes all sorts of peers, participants and 
members of involved communities and organisations such as students, 
potential students and also external analysts, mentors and experts. 
Nevertheless the perception, that there is no separation between teachers 
and learners any longer, because all participants are stakeholders of the 
same learning environment is not shared by all experts. According to that, 
some experts perceived a need to distinguish between teachers' and 
learners' peer production. 




Figure 12: Stakeholders of quality assessment in peer production 

and learning 2.0 approaches 
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E) How is the perception of the QMPP Process Model and QMPP 
Quality Matrix? 

QMPP Process Model 

In general the QMPP process model is perceived to illustrate a complete 
process whose steps give an adequate comprehensiveness, cover many 
aspects and are appropriate for diverse target audiences, such as teachers, 
researchers, students and other users. Nevertheless the experts noted that 
users who are not experienced and deeply involved in formal e-learning 
contexts, might find it difficult to apply the model to real life situations and 
therefore could need a more guided and argumentative approach in order to 
understand the process and optimize the implied steps. Concluding the 
model is expected to fulfil expectations concerning its usability in an 
experienced and well connected learning community, but needs to be 
supported by a social networking platform that creates a community 
environment and provides tools and guidelines to describe its steps and 
workflows in more detail, and thereby drive the inexperienced authoring, 
editing and evaluating users. Considering the fact that nowadays the 
process of learning on basis of peer production, as it is actually happening 
within environments such as Wikipedia and Facebook, is rather disruptive, 
unstructured and chaotic, experts think that the model either requires some 
sort of translation into a non static phase model or needs to motivate its 
potential users why to consider a redefinition of the production process. In 
accordance with these views the experts are afraid that the presented 
illustration of the process could discourage or even inhibit users. In 
particular the fact that some steps, such as "enriching” and "updating", are 
not clearly defined or differentiated could potentially create 
misunderstandings. Besides, there is a perceived need for a validating step 
in between the above mentioned phases of "enriching” and "updating”. 



QMPP Quality Matrix 

The QMPP Quality Matrix is perceived to be highly useful in terms of 
considering different viewpoints and modalities for the evaluation of the 
products to be managed. Highlighting the four most significant views and 
aspects of quality evaluation, the level of detail is considered to be useful 
and the scheme appears comprehensible, since it addresses the most 
familiar elements of formal as well as informal learning. Qn the contrary the 
fact that the concept does not differentiate between the process and the 
product of learning, which from the experts’ point of view demand individual 
parameters and different methods of research, is considered to be 
inappropriate and demands reconsideration. 
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Figure 13: Perception of the QMPP Process Model and QMPP Quality 

Matrix 



Summary of key points of quality related outcomes 

Facing the challenges of quality assessment and development of peer 
produced E-Learning content it is fundamental to put the attention on 
processes, behaviours and measurements of standardization and consensus 
building rather than outcomes or knowledge reproduction in order to 
achieve a supporting structure for the evaluation, coordination and 
systematization of creative, spontaneous and collaboratively created 
outcomes. Consequently It is essential to provide concepts, methods and 
Instruments that help peers to exploit the potential wisdom of crowds by 
means of social interaction and networks for the purpose of enabling a 
peer's personal growth, social integration and learning capabilities while 
gathering, combining and evaluating content with regards to its rigor, 
veracity and up-to-dateness as well as its usefulness and suitability for 
predefined objectives. With regards to supporting structures and indicators 
such as tools, guidelines and decision factors, the QMPP Process approach 
and Quality Matrix exemplify a comprehensive, appropriate and feasible 
framework that however needs to be a little bit more specific and guiding in 
certain aspects and requires to be embedded into community environment 
in order to be intelligible and applicable for every sort of stakeholder who Is 
participating in the process of learning, evaluation, and knowledge creation. 



4. 3. 2. 3 Overarching Areas 

a) Appraisal if evaluators of peer produced output should be co- 
designers of the output? 

The outcomes of the expert panels concerning the appraisal of the question 
if evaluators of peer produced output should be co-designers of the output 
are twofold. 
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First of all there is a perceived need to let e-learning products with a 
scientific background be evaluated by a responsible and competent scientific 
community. Concerning e-learning contents created within companies, the 
necessity to follow a set of steps is constituted. These steps however should 
adhere to companies' respective guidelines and publishing policies. 
Subsequently the consultation of a team of experts such as analysts and 
lawyers, who validate outcomes and workflows to drive content production, 
should complete the validation process. 

Beyond that the peer communities are required to define a set of 
parameters and criteria exemplarily based on instructional design models, 
before starting the process of creation and in order to adjust the peer 
production process and its outcomes to the particular subjects and the fields 
of development. According to the experts, this will help to diminish the 
potential for confusion and misunderstandings and thereby eventually 
provide a basis for smoothly proceedings of production. 

Finally it is agreed on the fact that the employed strategies should imply 
auto-evaluation and peer review -phases. Even though there also is a 
strong perception that the final validation often not occurs until the product 
is used by someone outside the group that created it. This final validation 
by individuals external to the group is considered to especially important in 
regards to assessing the attainment and achievement of specified goals and 
objectives of production. 



b) What are the strategies and processes for enhancing Peer 
Production? 

It is understood that in order to develop high quality peer produced e- 
learning content the participating peers need to be competent and well 
versed in the corresponding subject. This qualification is considered to be 
the key precondition for further measures and arrangements such as setting 
guidelines and defining a range of creativity. Accordingly the balance 
between knowledge/expertise and creativity is regarded to be the crucial 
factor and the main prerequisite for organizations which aim to add further 
strategies to enhance the quality of their peer produced content. In 
particular the experts named two other broad cognitions, identified within 
the educational context, which have the potential to improve the quality of 
outcomes. 

♦ The experts consider that a development motivated by a perceived and 
shared need inside the group accounts for positive effects on the 
peers' work ethics and working moral. Further on the outcomes of 
products developed out of a concrete need for learning are expected to 
have a fundamentally higher potential for future reuse. 

♦ The appliance of a "learning-team-centred" approach of course 
organization similar to those known from popular social networks and 
communities of practice, which utilize wikis and blogs as the primary 
way of interaction, is regarded to have an striking impact on the 
effectiveness of peer production. 
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c) How to deal with the risk, that including quaiity principies into 
peer production processes could negatively influence the 
potential for creativity and innovation? 

The degree of and potential for creativity is, according to the experts, highly 
depending on the context in which the production takes place. Comparing 
approaches of peer production within the educational contexts with those 
within corporate contexts, the extent to which peers can exert their 
creativity seems to be much more controlled and very limited in all the 
steps of the production process of corporate contexts. I n addition to that, 
the balance between creativity and rules Is considered to have differing 
impacts on different products. Accordingly products which have to follow 
strict procedures before being published naturally require more elaborated 
and stricter interpretation of rules and therefore consequently delimit the 
potential for creativity and spontaneous production. 

Summary of key points of overarching areas 

Regarding the assessment of outcomes by co-designers of the content it is 
highly important to distinguish between, and adjust in accordance to, the 
contextual background (scientific/corporate), field of development and 
particular subject of production in order to define, assign and modulate 
appropriate strategies, guidelines and criteria for working steps such as 
self-assessment, peer reviews, social recommendation, auto-evaluation and 
final validation which need to be dedicated to specific phases of evaluation. 
Besides empowering peers in terms of competence and qualification, an 
intrinsic motivation such as a perceived shared need for learning inside the 
group and a "learning-team-centred" approach such as those utilised by 
social networks and communities of practice as Wikipedia, are considered to 
be effective strategies with a strong and enduring impact on the 
enhancement of peer production. Further on the feasibility of creating a 
balance between creativity and rules Is not only considered to be greatly 
depending on the context (educational/ corporate) of production but also to 
have highly differing Impacts on different products and In the case of the 
strict publishing procedures applied in the corporate context, are definitely 
delimiting the potential for creativity and spontaneity. 
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ANNEX 1: 


EXPERT PANEL REPORTS 



Annex 1.1: Expert Panel Report - Germany 



General Description of the Expert Panels 



Name of the responsible 
QMPP partner organisation; 


EFQUEL (test panel in Innsbruck) 


Name of the responsible 
rapporteur(s); 


Ulf Ehlers, David Riley, ... 


Name of the hosting 
organisation: 

(in case different fronn the 
responsible QMPP partner 
organisation) 


EFQUEL ^ EFQUEL workshop 


Location of the verification 
meeting: 

(please report country and 
city) 


1 nnsbruck 


Date of the Expert Panel; 


25 JUNE 2008 


Number of participants; 


15 


Profile of participants: 
(please include nannes of 
involved QMPP partners and of 
external panellists; please add 
as nnany rows as necessary) 


Educational Experts 
Teachers (HE, School, VET) 
E-Learning Experts 



# 


Last 

name 


First 

name 


Name of 
organisation 


Role within 
organisation/ 
Expertise with 
intergenerational 
learning 


Contact details 


1 












2 












3 












4 












5 













Format of the verification 
process; 


X 1 = Learning Cafe; 

□ 2 = Reflective Workshop/Focus Group Discussion; 

□ 3 = Audio/Web Conference. 


Means of contact; 


X 1 = Face-to-face Meeting; 

□ 2 = Telephone Interview; 

□ 3 = Online Consultation. 


Agenda of the Expert 
Panel; 

(please add a detailed 
agenda of the expert 
meeting) 


See attached, in line with the suggested agenda for 
expert panels 


General 

description/ aspects of 
the Expert Panel; 
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Outcomes of the Expert Panels 



(please describe the different elements and criteria as detailed as possible) 



O What is the object of 
the quality assessment? 


• Process of learning, communication and interaction rather 
than outcomes of learning 

• Behaviour rather than knowledge reproduction 

• social networks as assets of social interaction can be 
assessed for quality 


© What are the 

dimensions of the 

quality assessment? 


• Personal growth and development of own individual 
learning capabilities 

• The social network which as been created has to be taken 
into account because web 2.0 learning focuses on this. 

• The group processes and the “social system" has to be 
assessed. 


© What are methods 
and instruments to 

assess/develop quality? 


• to assess outcomes 

• A problem is that learning in web 2.0 scenarios takes place 
in a serendipitious manner and can not be assessed from 
pre-defined standards. 

• A problem is that material if it is developed by the 
stakeholders and not the experts might not be correct. 

• In web 2.0 scenarios the "crowds" have more wisdom than 
th individual expert. This wisdom needs to be taped into. 

• There might be a garbage in garbage out problem because 
there is now correcting expert?! 

• Quality development in web 2.0 learning demands for a 
consensus and an empowerment of the learners to assess 
and evaluate their own learning. It is not always 
compatible with institutional guidelines and regulations. 


O What are the 

stakeholders of the 

quality assessment? 


• Everybody participating in the social process 

• There is no separation anylonger between teachers and 
learners, all are stakeholders of the learning environment 


Additional Aspects 


QMPP Process Model 


QMPP Quality Matrix 


Usefulness 






Usability 






User-friendliness 






Comprehensiveness 


It seems to be 
comprehensive an covers 
many aspects 




Comprehensibility 


Difficult to connect to the real 
life - how can it be applied? 




Appropriateness 

(for different target 
groups) 


The problem is that is 
suggests a structured and 
ohased approach for a 
potentially chaotic process of 
disruptive learning processes. 
Web 2.0 learning can not be 
analysed with static phase 
models. Therefore th model 
needs translation 




Recommendations 

(for necessary 
improvements) 






Additional comments 
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Annex 1.1.1 Outline of the Expert Panel - Germany 

A) Short I ntroduction to QMPP 

0 Rationale, objectives, envisaged outcomes 

0 Web 2.0 Technologies e.g. blogs, wikis, podcasts, social 
bookmarking, personal learning environments, e-portfolios 

0 Peer production 

0 Quality assessment 

0 Initial and Continuous Vocational Education and Training (I -VET, 
C-VET), Continuous Professional Development (CPD), skills, 
competences 

0 I ntroductory questions: 

a) Is quality in peer production processes with Web 2.0 applications 
manageable? 

b) Is quality manageable with traditional approaches (e.g. ISO, 
EFQM)? 

c) Is there a generic process model of peer production? 



B) Presentation of 3 QMPP Practice Cases 

Short presentation of 3 QMPP practice cases e.g. lAVANTE, 
www.azubi.net . Finish Elevator Company (optional) 



C) Discussion I : Case Assessment & Suggestions for I mprovement 

Three Speed Cafes on the three best ways to develop / to improve the 
quality of peer production in the 3 QMPP practice cases. 



D) Presentation of Changes from Web 1.0 / E-Learning 1.0 -> Web 
2.0 / E-Learning 2.0 



E-Learning 1.0 


E-Learning 2.0 


Quality assessed through 

experts 


Quality assessed through learners and 
peers 


Learning platform 


Personal Learning Environment 


Content 


User Created Content 


Curricula 


Learning diaries/ e- portfolios 


Course structure 


Communication 


Tutor availability 


1 nteraction 


Multimedia (Interactivity) 


Social networks / Communities of 
Practice (CoP) 


Acquisition processes 


Participation processes 
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E) Presentation of the Process Model of Peer Production 




« Creating 

0 (shared) authoring of courses, texts, resources, wikis; creating 
images, audio materials, video materials etc. 

« Validating 

0 validating content with subject matter experts, vaiidating content 
with peers, rating the content etc. 

« Editing 

0 shared editing, undertaking peer reviews, creating alternative 

navigational routes etc. 

« Enriching 

0 creating additional content materials, wikis, publishing individual 
works and team works, sharing or learning (b)logs, social 
bookmarking etc. 

« Updating 

0 monitoring existing content, updating existing content, adding 

specific area content etc. 

« Benchmarking 

0 identifying of good cases and practices, identifying of good digital 
resources, sharing iearning experiences by sharing iearning 
(b)logs etc. 
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F) Presentation of the Peer Production Quaiity Matrix & reiated 
Research Questions 



O What is the object 
of the quality assessment? 

• Learning process 

• Work/business process 

• Degree of 

communication/ participatio 
n 

• Learning 
achievement/outcomes 

• Learning object, course 
content, resource 

etc. 

0 What are methods and 

instruments 

to assess/develop quality? 

• Self assessment 

• External assessment 

• Peer review 

• Collaborative dialogue 
etc. 



0 What are the dimensions 

of the quality assessment? 

• Pedagogical 

• Technological 

• Economical 

• Institutional 

• Organisational 

• Cultural 
etc. 



O What are the stakeholders 

of the quality assessment? 

• Users 

• Experts 
etc. 



G) Discussion II: Benchmarking the Quaiity Assessment in Peer 
Production Processes with Web 2.0 Applications 

Four Learning Cafes on how to benchmark the quality assessment of 
peer production processes with Web 2.0 applications by relating the 
Peer Production Quality Matrix with the Process Model of Peer 
Production 



H) Rapport from the Learning Cafes & Final Plenary Discussion 
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Annex 1 . 2 : Expert Panel Report - Italy 



I ntroduction 

The objective of the Work Package 3 'Benchmarking Peer Production' is to 
identify key approaches of quality management in peer production of e- 
Learning content by benchmarking peer production practices and processes 
in other areas (such as the creation of technical documentation, joint 
editing efforts etc). The key activities include structured benchmarking of 
other areas of peer production of digital content and the organisation of 
three Expert Panels, which are aiming at exploring the experiences in peer 
production mechanism, processes and practices. 

The purpose of the present document is to summarise the outcomes of the 
Expert Panels in Italy on benchmarking approaches for the peer production 
of (digital) content (as developed within Work Package 2 and 3). 

The report is structured in three chapters: 

a. the first chapter is aimed at describing the organisation of the expert 
panel and the profile of the Italian expert panel. 

b. The second chapter Illustrates the results emerging from the 
completion of QMPP questionnaire and individual interviews. 

c. The third part presents some conclusions derived from the consultation 
of the Italian expert panel. 



Organisation and description of the Italian expert panei 

The selection of the experts to be included in the panel had been made by 
Pier Giuseppe Rossi (university of Macerata) and Claudio Dondi (Scienter) 
taking in consideration the agreed decisions made in the previous steps of 
projects and the need to make a choice consistent with them; two main 
issues were considered: 

♦ the necessity to identify a proper number of experts to be effective for 
the specific aim: following the directions agreed in the Bologna 
meeting and in order to have a homogeneous overview among the 
panels in Germany, Spain and Italy the number of Italian experts has 
been defined to be seven; 

♦ the relevance to focus on different backgrounds: with regards to the 
previous project activity for WP2 in which partners presented and 
described different case studies from three main contexts (higher 
education; corporate; vocational training) and in order to offer a wide 
range of viewpoints Italian experts were chosen from both the 
educational field and the corporate context. 

The organization of the panel required a preparatory phase which occurred 
before the summer break when experts were individually contacted by e- 
mail to check their availability and schedule a meeting. 
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Due to the summer period in Italy it was not possible to organize a F2F 
meeting and since the delay should have been too long to be able to gather 
data in the due time for the synthesis report Michela Moretti (Scienter) and 
Laura Fedeli (Universita di Macerata) organized the panel in two phases 
which iet experts contribute individualiy at a distance: 

♦ a questionnaire has been sent via e-mail (see annex I) along with a 
descriptive brochure of the Project that was translated into Italian to 
be better accessed by the nationai experts; 

♦ once received the questionnaire back a phone interview was scheduled 
to deepen open questions (see annex II). 



Description of Expert panei 

As you can see from the table below the seven experts cover different 
professional contexts whose range goes from the public institution 
(Universities, national research centers) to the private corporate of different 
size from little-sized ones to large corporate companies. 



# 


Last 

name 


First 

name 


Name of 
organisation 


Role within 
organisation 


Contact details 


1 


Amicucci 


Franco 


Amicucci 

Formazione 


Legal 

representative 


af@amicucciformazione.com 


2 


Colorni 


Alberto 


Politecnico 

Milano 

Centro METI D 


President 


alberto.colorni@polimi.it 


3 


Pescuma 


Saverio 


ISFOL 


researcher 
and European 
project 
manager 


s.pescuma@isfol.it 


4 


Renesto 


Ivan 


Microsoft 

Italy 


Education 

Technology 

Advisor 


i-ivanr@microsoft.com 


5 


Ravotto 


Pierfranco 


CIDI Milano 


Expert 


pierfranco.ravotto@gmail.com 


6 


Simone 


Aurelio 


Universita 
Roma Tor 
Vergata 


Director lAD; 
president of 
Siel (Italian 
society of e- 
learning) 


direttore@scuolaiad.it 

presidente@sie-l.it 


7 


Trent! n 


Gugliemo 


CNR Genova 
1 nstitute for 
Educational 
Technology - 
Italian 
National 
Research 
Council 


Senior 

researcher 


trentin@itd.cnr.it 
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Results from the consultation of the Italian expert panel 

1. Defining the kev characteristic of a peer 

To define the characteristics of a "peer" experts focus on the attitudes that 

wouid be criticai, as underiined in the foiiowing viewpoints, a peer has to: 

♦ be open-minded and be abie to iisten to others; 

♦ possess strong reiationai skiiis and auto-criticai skiiis, 

♦ show a good reiationship management, that is the skiiis to enroii 
peopie towards mutuaiiy beneficiai goais, 

♦ have a deep subject matter competence in a specific sector 

♦ be aware of what being part of a community means: kind of interaction 
which can occur (not to be Just a iurker but be an active member and 
react to stimuii), times of interaction (the exchanges among peers 
have to be quick and prompt) 



2. Defining the meaning of "peer group/ community" 

The definition of peers' characteristics is strictiy reiated to the concept of 

peer community, experts reported their vision of a peer group underiining 

the foiiowing aspects: 

♦ Different communities (e.g. students, teachers) require a different 
definition but generaiiy speaking in a peer group members are wiiiing 
to share materiais, re-edit existing ones and create knowiedge, they 
have a ciear and expiicit objective to support each other to grow 
together; 

♦ There’ no hierarchy for what concern roies among members but 
converseiy a hierarchy determined by the confidence acquired from 
experience and competencies of some members, this kind of authority 
is generaiiy naturaiiy recognized by the group; 

♦ the openness towards the other communities and reaiities is reievant 
even if the range of it is conditioned by the specificainess of the 
subject the group deai with; 

♦ the group necessariiy recognizes a set of shared communication and 
operationai modaiities (impiicit for the members); 

♦ within the group there can be a range of ieveis of members' 
participation (from the iurker to one who contributes just if fostered 
and the one who spontaneousiy offers his/her contribution such as 
materiais, know-how because perceive the reievance for the group); 

♦ when we speak about "sharing" it’s proper to underiine that aiso 
iearning paths and projects can be shared by members and not oniy 
singie materiais (e.g iearning unit as L.O.) 
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3. Defining the meaning of PEER PRODUCTI ON 

From the definition of peer production provided in the questionnaire: "the 

digital content created, edited, enriched by peers, in other words by people 

on the same hierarchical level" experts underlined a variety of inputs: 

♦ the need to distinguish between teachers' peer production and 
learners' peer production. In the latter case the definition fits well but 
In the former case It wouldn't be appropriate to stick to digital contents 
only and would be better to extend the concept to the structure 
(project) of educational modules and/or whole courses, or at least to 
the path proposed to the learner; 

♦ the relevance of recognizing the value of a complex process within 
production (creation, sharing and editing) and the need to focus on the 
perceived need of "peers" to share the same filed of interest and 
significant objectives as the starting point to stimulate production; 

♦ The process of creation/production requires the peer group to show an 
initial openness towards the inputs that can arise from different 
directions; 

♦ The risk hidden inside the expression "hierarchical level": it seems to 
refer only to people of the same organisation. It would be better to 
highlight that often the collaboration is amongst people who are not 
colleagues in the same organisation but that could also be suppliers if 
we deal with companies. 



4. Learning from EXPERIENCE: peer production developed within 
experts' institution 

We can summarize experts' direct experience by splitting the data they 
reported into two main areas: 

1. contexts in which e-iearning products were created: 

0 educational activities in educational technology teachers' training 
0 post graduate courses when it is required a final project work 
0 collaborative production of LOs by teachers in school projects 
(www.sloopproJect.eu, Sharing Learning Objects in an Open 
Perspective; Sir 2 : www.tes.mi.it/sir 2 portale) 

0 few examples of production of learning contents by students 
0 new experimental models in ICT use must be tested within 
research activities of the institute; 
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2. tools used for the production, development and editing: 

Since most ITD e-learning peers activities concern the use of 
networked collaborative learning strategies the tendency is to use 
digital material which is not necessarily structured and designed 
according to ODL criteria; rather than designing actual e-learning 
contents there's a tendency to collaborate In the design and reuse of 
e-learning based processes: 

0 LCMS (Noodle) and platforms in which it is possible to develop 
collaborative work(C.S.C.W); 

0 web 2.0 applications (delicious, flickr, youtube; environments 
such as wiki, weblogs; e-mail/mailing list; web forum); 

0 HTML editors (like GoLive or DreamWeaver); 

0 L.O. editors (eXe, Reload), applications enabling you to create 
interactive exercises (HotPotatoes), logical maps (CMAP); 

0 In the case of Microsoft Italy specific applications for the 
production of multimedia contents are used such as "Learning 
Essentials 2 . 0 " for Microsoft Office; THESIS e-learning product 
suite 



5. Products/ artifacts created bv peers in the above mentioned 
experiences 

The final products appear to be mainly: 

♦ LOs dealing with different subjects (mostly when the contents are 
related to computer technologies, fields where the knowledge changes 
quickly); 

♦ the storyboards of the above mentioned learning object; 

♦ articles, book chapters, Internet resources and sometimes materials 
produced by learners during course activities (wiki, problem solving, 
reports, documents etc.); 

♦ tutorials (PPT/Flash) about for example "How to use Reload", "How to 
use Rss", "How to J avaScrIpts" 

♦ With regard to the reuse of those products experts underlined the 
following aspects: 

♦ the "littler" are the learning units produced the more they can be 
reused and be made available for other paths; 

♦ besides contents it's possible to reuse a model, a course structure as 
for example some learning paths developed within Noodle; 

♦ the material developed by students during courses, it is often made 
available to participants of successive courses: 

0 as material to be added in a shared repository; 

0 As complementary educational material (appropriately revised by 
the teacher); 

0 as material for discussions and comparison. 
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6. Process Model of Peer Production 



The general perception of the QMPP process model is that appears as a 
complete process whose steps give an adequate comprehensiveness; it 
seems to be appropriated for different target audience, not only teachers 
and researchers (with pedagogical skills) but could also be implemented for 
students or other users. Even though almost all experts stated that the 
model show a good comprehensibility there's a concern about users who are 
not deeply involved in formal e-learning contexts, they, in fact, could need a 
more guided and argumentative approach to reaily understand the process 
and optimize ali the steps. 

The entire process seems to be planned to work perfectly in a Connected 
Learning Community, shows a good usability, but somehow conditioned by 
the presence of a social networking platform to create a community 
environment, templates and wizards to drive authoring users, workflows 
describing the process model. 

Being the peer production practices underiined in the project (Wikipedia, 
Facebook, etc) mostly informal, spontaneous and etero- directed it would be 
good to have guidelines to inform users about the usefulness to set a 
redefinition of the production process otherwise the modei could be 
perceived as a distracting eiement rather than a fruitful one. 

The formalization of the process in the way it's presented could inhibit 
users, demotivate and disorient them, besides some steps are not clearly 
definite and may create misunderstandings iike the difference between the 
steps "enriching" and "updating". 

It's also recommended to clarify that within those steps a validating step is 
needed. 



A. PEER PRODUCTI ON QUALI TY MATRI X 

The matrix reveals as highly useful since the evaluation of the product 
undoubtedly has to be managed taking in consideration different viewpoints 
and modaiities. The modei is detaiied and its four components underiine the 
main aspects and actors for a quality evaluation. 

The whole scheme is very comprehensible since it's connected to elements 
of both formal and informal learning that are nowadays very famiiiar. 
Nevertheless with regard to the appropriateness it seems a little reductive 
to tie the same list of evaluation (1) to both the process and the product 
which would require individual parameters as well as different methods of 
research that are not necessarily overlapping. 
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B. FACING COMPLEXITY: THE EVALUATOR OF THE QUALITY OF A 
PEER PRODUCED OUTPUT IS ALSO A CO-DESIGNER OF THE 
OUTPUT 

Two main aspects arose from the interviews with the experts: the need to 
make the products be evaluated by the scientific community and for those 
e-learning contents created in companies, the need to follow a set of steps: 
adhere to company guidelines (publishing policies), consult a team of 
experts who validate contents and after lawyers' validation a workflow 
drive contents to complete the validation process. 

The peer community should set the parameters and criteria (e.g. from 
instructional design models) before starting the process of creation 
according to the subject and the field. In this way there won't be space for 
confusion and the process would proceed smoothly. 

The strategies to set should imply an auto-evaluation and a peer -review 
phase during the whole process but there's also the perception that the final 
validation occurs when the product is being used, it's the one who reuses 
that product (outside the group that created it) who can actually check its 
quality and one of the parameter to consider is that the product to be 
valuable should finally reach the due objectives. 



C. STRATEGY AND PROCESSES FOR ENHANCING PEER 
PRODUCTI ON 

The necessary condition for a peer group to be able to produce content of 
quality is to have the due competencies in a specific subject, just in this 
case the same peer can set their guidelines and also define the range of 
creativity. 

Creativity and subject matter experience (and their balance along the 
production process) can be the key elements for an organization to foster 
peer- production, some of the strategies that have been identified in the 
educational context (school, universities) and can be applied to enhance the 
production are: 

♦ the creation of a product has to develop from a perceived and shared 
need inside thegroup: for example during a university course slower 
students may need to be helped In the comprehension of rough 
materials and more expert colleagues can be fostered to produce 
tutorials or additional study material to be accessed by those peers; In 
this case the same products could be easily reused In the following 
course since the were born from a concrete learning need; 

♦ The whole organization of the course has to have a learning-team- 
centred approach similar to those that refer to community of practice 
and social networks and which utilize wikis and blogs to interact. 
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D. QUALITY VERSUS CREATIVITY/ I NNOVATI ON: RISK OF 

INCLUDING QUALITY PRINCIPLES IN A PEER PRODUCTION 
PROCESS 

Different contexts show a highly different approach towards creativity. 
When we compare educational context like schools projects and context like 
corporates we find that in the latter the control over creativity is strong and 
present in all the steps of the production process. 

The balance between creativity and rules seems to have a relevant value for 
quality mostly when we deal with specific products that have to follow strict 
procedures before being published. 



Conclusion 



To conclude we would like to highlight the interests Italian experts showed 
in the research area and in the direction the QMPP Project gave to 
development of the topic. 

Even though the modality we were forced to choose to conduct the panel 
(due to summer break and short times) is not the most fruitful way to 
gather data for two main reasons: 

♦ the inquire was run individually (with no interaction among experts); 

♦ the used channels can't offer such a wide opportunity to discuss ideas, 
doubts, suggestions 

we are very satisfied with the inputs we received and glad to confirm that 
the wish to offer a more productive contribution and the awareness of the 
relevance of the research made the experts explicit their availability for a 
future commitment along the project to share their experience and provide 
their support in the validation process. 

Besides, one of the most relevant aspects we believe it could be a source of 
reflection among the partnership is the general concern perceived by the 
experts about the need to define a quality system that is able to avoid a too 
rigid process which could inhibit the production. 
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Annex 1.2.1 Questionnaire for the ITALIAN Expert 
Panels 



Profile 



Last name: 

First name: 

Name of organisation: 
Role within 
organisation/Expertise with 
intergenerational learning: 

Contact details: 



QUESTIONS: PEER PRODUCTION 

1. Peer production: the digital content created, edited, enriched by 
peers, in other words by peopie on the "same hierarchical level". 

Does the above definition need to be modified or enriched? What is 
your definition of peer production? 



2. I s peer production present in your work context? If yes, how has it 
been developed by your institution? 
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3. I n case you activated strategies in order to foster peer production in 
your context 

D A What procedures did you follow to create e-learning contents? 



□ 



B What procedures do you follow for checking or for the validation 
of contents? 



D C Which e-learning tools do you use? 



4. Describe the products/ artifacts created by peers in the above 
mentioned experiences 



5. How were your products used/ reused? 
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QMPP PROPOSAL 



1. Presentation of the Process Model of Peer Production 



♦ Creating - (shared) authoring of 
courses, texts, resources, wikis; 
creating images, audio materials, 
video materials etc. 

♦ Validating - validating content 

with subject matter experts, 

validating content with peers, 
rating the content etc. 

♦ Editing - shared editing, 

undertaking peer reviews, 

creating alternative navigational 
routes etc. 




Enabling policies 



Enabling proDedures 



Enabling processes 



Enabling 1mls 



♦ Enriching - creating additional 
content materials, wikis, 
publishing individual works and team works, sharing or learning 
(b)logs, social bookmarking etc. 



♦ Updating - monitoring existing content, updating existing content, 
adding specific area content etc. 

♦ Benchmarking - identifying of good cases and practices, identifying 
of good digital resources, sharing learning experiences by sharing 
learning (b)logs etc. 



1.1 Research Questions 





QMPP Process Model 


Usefulness 




Usability 
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QMPP Process Model 


User-friendliness 




Comprehensiveness 




Comprehensibility 




Appropriateness 

(for different target 
groups) 




Recommendations 

(for necessary 

improvements) 




Additional comments 
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2. Presentation of the Peer Production Quality Matrix 



o 


0 


What is the object 


What are the dimensions 


of the quality assessment? 


of the quality assessment? 


• Learning process 


• Pedagogical 


• Work/business process 


• Technological 


• Degree of 


• Economical 


communication/ participation 


• Institutional 


• Learning achievement/outcomes 


• Organisational 


• Learning object, course content, 


• Cultural 


re source 


• etc. 


• etc. 




0 


o 


What are methods and 


What are the stakeholders 


instruments 


of the quality assessment? 


to assess/develop quality? 






• Users 


• Seif assessment 


• Experts 


• Externai assessment 


• etc. 


• Peer review 




• Coilaborative dialogue 




• etc. 





2.1 Research Questions 





QMPP Quality Matrix 


Usefulness 




Usability 
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QMPP Quality Matrix 


User-friendiiness 




Comprehensiveness 




Comprehensibiiity 




Appropriateness 

(for different target 
groups) 




Recommendations 

(for necessary 

improvements) 




Additional 

comments 





Thank you for your contribution! 
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Annex 1 . 2 . 2 : Italian Expert Panels grid for interview 



Reference questions used in the interview 



1. Deepen the definition of "peer" focussing on your own work context 

2. Which characteristic does a group have to show to be considered a 
"peer community"? 

3. Specify the experiences in which peer production occurred and how the 
artifacts were reused 

4. How does the evaluation of peer production have to be managed, 
which dynamics does it develop, which methodologies and tool does it 
need to be activated?) 

5. Which are the processes that can foster the quality in the peer 
production ? 

6. Can the introduction of methods to evaluate the quality of peer 
production inhibit creativity and reduce its potentialities? 
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Annex 1 . 3 : Expert Panel Report - Spain 



General Description of the Expert Panels 



Name of the responsible 
QMPP partner 
organisation; 


SCIENTER Espaha & lAVANTE 


Name of the responsible 
rapporteur(s); 


Miguel Angel Munoz 


Name of the hosting 
organisation: 

(in case different fronn the 
responsible QMPP partner 
organisation) 


lAVANTE 


Location of the verification 
meeting: 

(please report country and 
city) 


lAVANTE Granada (CMAT) 

Pargue Tecnologico Ciencias de la Salud 
Granada (SPAI N) 


Date of the Expert Panel; 


September 2008 


Number of participants; 


9 


Profile of participants: 


# 


Last nanne 


First 

name 


Nanne of 
organisation 


Role within 
organisation 


Contact details 


1 


Castro 


Carlos 


Universidad 
de Granada 


He is 

teacher of 
School of 
Library and 
1 nfornnation 
Science, but 
he have 
been 

responsible 
for 10 years 
of the 
regional 
policies of 
1 nfornnation 
Society in 
Extremadura 


Carlos. anulinexCcDgmail. com 


2 


Lopez 

Calero 


Mabel 


Fundacion 
l+D del 
Software 
Libre 


Technical 
Manager in 
this 

foundation 
for research 
in the open 
source 
software 


milooezCcDfidesol.ora 
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3 


Soledad 


Puentes 


Consorcio 
Fernando 
de los Rios 


Technical 
Manager in 
this public 
consortium 
responsible 
of the 
Guadalinfo 
network 
(public 
network of 
ITC centres 
of resources 
in 

Andalusia) 


Soledad. fuentesCcDi utnadeandalucia.es 


4 


Sebastian 


Torres 


Grupo 
TADEL / 
ESEA 


eLearning 

Responsible 


storesCcDaruDotadel.com 


5 


Purificacion 


Ceron 


lAVANTE 


Responsible 
of eLearning 
Programs 


DuriceronCcDiavante.es 


6 


Sabas 


Casas 


1 ntecna 
Soluciones 


General 

Director 


scasasCcDintecna.es 


7 


Miguel 


Gea 


CEVUG 


Director 


maeaCcDuar.es 


8 


Oscar 


Martin 


CEVUG 


Design 

Technician 


omartin(cDuar.es 


9 


Emilio 


Arjona 


CEVUG 


eLearning 

Technician 


emilioCcDuar.es 


Format of the verification 
process; 


□ 1 = Learning Cafe; 

0 2 = Reflective Workshop/Focus Group Discussion; 

□ 3 = Audio/Web Conference. 


Means of contact; 


0 1 = Face-to-face Meeting; 

□ 2 = Telephone Interview; 

□ 3 = Online Consultation. 


Agenda of the Expert 
Panel; 

(please add a detailed 
agenda of the expert 
meeting) 


Welcome and short introduction to QMPP 
Presentation of the experts 
Setting the Scene 

Presentation of the experts' experiences 
Definition of Peer Production 

Presentation of the Process Model of Peer Production & 
Discussion 

Presentation of the Peer Production Quality Matrix & Discussion 
Conclusions 


General 

description/ aspects of the 
Expert Panel; 


The selection of the expert had been made by Scienter Espaha 
and lAVANTE in the Andalusia area in order to make a 
presential session. 

They have been involved not only for this expert panel, but as 
Advisory Group for all the phases of the QMPP project. 

There was selected 9 organisation: 

Mr. Carlos Castro was seleted for his personal experience 
as responsible of a regional strategy to promote the 
Information Society in the Extremadura region. 

Mrs. 1 sable Perez, responsible of different initiatives for the 
development of learning material for foreign languages 
through teachers communities. 

Fundacion I-fD del Software Libre, a professional 
community for development in the open source area 
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especially in JAVA. 

Consorcio Fernando de los Rios, as responsible of the 
Guadalinfo network, that is trying to create a professional 
knowledge repository of their connnnunity of coordinators of 
the Guadalinfo Centres. 

Grupo TADEL / ESEA, training Group active in the elearning 
area. 

People who work in the professional connnnunities of 
lAVANTE 

Intecna Soluciones, an innovative software house in the 
Health Technological Park of Granada. 

CEVUG, responsible of the digital strategies of the 
University of Granada. 

Portal EVA, responsible of the training strategies of RETA, 
the Andalusian Network of Technological Areas, the nnost 
important community of innovation enterprises in the 
Region. 

Finally Mrs. Perez ad Portal EVA expressed their impossibility to 
participate in this meeting, but they showed interesting to be 
involved in future activities of this advisory group. 

The CEVUG asked about to involved 3 people because they are 
working in different strategies of scientific peer production 
communities and they are very interesting in the results of this 
project. 

INTECNA, that doesn't worked specifically in the learning area 
expressed that they are very interesting in any process to 
make tacit the production of knowledge of their professionals. 
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Outcomes of the Expert Panels 

After presents the different experiences of the Group, the expert agree the 
wide approach of the QMPP project, expressed in the matrix with the critical 
dimensions of setting of the objectives and the structure. 

But a central question that the expert asks to the project to better clarifies: 
When a digital resource o material created through a peer exchange can be 
considered a "learning content"? There was peer production process that 
finishe in a common knowledge, but not in a real product. 

For the rest, the group agrees the usefulness, usability, user-friendliness, 
comprehensiveness and comprehensibility of the QMPP process model. 
Quickly, the group has opened the discussion about aspect of the Quality 
Matrix, so the coordinators of the session had decided to propose the same 
activity that was used in the Microlearning Conference. 

The result of the activity was: 
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O What is the object 
of the quality assessment? 

• Evaluation against Coordination 

• Systematization of spontaneous knowledge 
building processes 

• Consensus building: 

o Awareness (peer reviewers) 
o Trust 
o Cohesion 
o Participation 

o Criteria to achieve the consensus 

• Behaviours of: 
o Peers 

o Potential Students 
o Community 

o Producers of the learning material 

• Organisation of the peer group 

• Oriented to results: 

o Typology of contents 
o Ways to record and tag the contents 

• Achieve of the proposed objectives: 
o Measuring instruments 

o Level of satisfaction of users 

• Methodology 

• Planning and control of activities 

• Communication process 

• Usefulness of contents: 

o Effectiveness - Measuring of knowledge 
o Standardisation of the contents 



© What are the 

dimensions 

of the quality assessment? 

• Motivation and 
recognition 

• Freshness - updating 

• Rigor and Veracity of 
results 

• Content < = > Process 
to arrive to the content 

• Clear definition of the 
context: 

o Characteristic of 
peers 

o Learning area 

(vocational, 
professional, 
educational,...) 
o Open or close 
environment 

• Peer group: 

o Satisfaction with 
process 

o Satisfaction with 
results 

• Final users: 

o Satisfaction 
o Adaptation 

• Dimension of the peer 
group 

• Instruments and tools: 
o Suitability 

o Competences of the 
group (sometimes 
the better expert 
group is not able to 
use ICTs) 
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e What are methods and instruments 

to assess/develop quality? 

System of indicators: 

o Records available in tools -> Attention to 
infrastructures and software 
Pools, surveys and questionnaires 
External audit: 

o Other peers - Way to selected: 

■ Random selection 

■ Voluntaries 

■ Reference group 

o Experts 

Guidelines -> Decision factors 
Competencies: Initial training of the peer 
group 



O What are the 
stakeholders 

of the quality assessment? 

• Peers: 

o Particpants 
o External 

• Potential Students 

• Community / 

Organisation 

• Producers of the 
learning material 
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Annex 1 . 3 . 1 : Images of the Expert Peer Production 
Quali ty Matri X 
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ANNEX 2: Reporting Template and Expert Panel Outline 
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1. About this Document 

The purpose of the present document is to summarise the outcomes of the 
Expert Panels in three European countries i.e. Finland, Italy and Spain on 
benchmarking approaches for the peer production of (digital) content (as 
developed within Work Package 2 and 3). 

The objective of the Work Package 3 'Benchmarking Peer Production’ is to 
identify key approaches of quality management in peer production of e- 
Learning content by benchmarking peer production practices and processes 
in other areas (such as the creation of technical documentation, joint 
editing efforts etc). The key activities include structured benchmarking of 
other areas of peer production of digital content and the organisation of 
three Expert Panels, which are aiming at exploring the experiences in peer 
production mechanism, processes and practices. 

In order to guide the discussions an outline for the Expert Panels has been 
produced, which sequences the following activities or elements (see Annex I 
for more details): 

A) Short I ntroduction to QMPP; 

B) Presentation of 3 QMPP Practice Cases; 

C) Discussion 1: Case Assessment & Suggestions for Improvement; 

D) Presentation of Changes from Web 1.0 / E-Learning 1.0 Web 2.0 / 
E-Learning 2.0; 

E) Presentation of the Peer Production Quality Matrix & related Research 
Questions; 

F) Discussion II: Benchmarking the Quality Assessment in Peer 

Production Processes with Web 2.0 Applications; 

G) Reporting from the discussions. 

Furthermore the QMPP partnership agreed on three different formats for the 
Expert Panels: 

♦ Learning Cafes; 

♦ Reflective Workshops/ Focus Group Discussions; 

♦ Audio/Web Conference. 

♦ The report is divided into two chapters: the first chapter provides a 
general description of the Expert Panels: 

♦ Responsible partner organisation, name of rapporteur(s), name of 
hosting organisation; 

♦ Place, date and timing of the meeting; 

♦ 1 nvolved experts; 
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♦ Applied approach (i.e. Learning Cafes, reflective workshop/focus group 
discussion, audio/web conference); 

♦ Means of contact i.e. face-to-face/telephone/online; 

♦ Agenda of the meeting; 

♦ General description/aspects of the meeting. 

The second chapter gives then a summary of Expert Panels according to the 
four main elements of the QMPP Quality Matrix: 

♦ Object of the quality assessment; 

♦ Dimensions of the quality assessment; 

♦ Methods and instruments to assess/develop quality; 

♦ Stakeholders of the quality assessment. 

In addition the QMPP Process Model and QMPP Quality Matrix should be 
briefly assessed according to the following criteria: 

♦ Usefulness; 

♦ Usability; 

♦ User-friendliness; 

♦ Comprehensiveness; 

♦ Comprehensibility; 

♦ Appropriateness (for different target groups); 

♦ Recommendations (for necessary improvements). 
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2. General Description of the Expert Panels 



Name of the responsible 
QMPP partner 
organisation: 




Name of the responsible 
rapporteur(s): 




Name of the hosting 
organisation: 

(in case different from the 
responsible QMPP partner 
organisation) 




Location of the verification 
meeting: 

(please report country and 
city) 




Date of the Expert Panel: 




Number of participants: 




Profile of participants: 

(please include names of 
involved QMPP partners and 
of external panellists; please 
add as many rows as 
necessary) 




# 


Last 

name 


First 

name 


Name of 
organisation 


Role within 
organisation/ 
Expertise with 
intergenerational 
learning 


Contact details 


1 












2 












3 












4 












5 












Format of the verification 
process: 


□ 1 = Learning Cafe; 

□ 2 = Reflective Workshop/Focus Group Discussion; 

□ 3 = Audio/Web Conference. 


Means of contact: 


□ 1 = Face-to-face Meeting; 

□ 2 = Telephone Interview; 

□ 3 = Online Consultation. 


Agenda of the Expert 
Panel: 

(please add a detailed 
agenda of the expert 
meeting) 




General 

description/ aspects of 
the Expert Panel: 
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3. Outcomes of the Expert Panels 

(please describe the different elements and criteria as detailed as possible) 



O What is the object of the 
quality assessment? 




@ What are the dimensions 

of the quality assessment? 




© What are methods and 
instruments to 

assess/ develop quality? 




O What are the 

stakehoiders of the 

quality assessment? 




Additional Aspects 


QMPP Process Model 


QMPP Quality Matrix 


Usefuiness 






Usabiiity 






User-friendiiness 






Comprehensiveness 






Comprehensibiiity 






Appropriateness 

(for different target groups) 






Recommendations 

(for necessary 
improvements) 






Additionai comments 
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Annex 2 . 1 : Outli ne of the Expert Panels 

A) Short I ntroduction to QMPP 

♦ Rationale, objectives, envisaged outcomes 

♦ Web 2.0 Technologies e.g. blogs, wikis, podcasts, social bookmarking, 
personal learning environments, e-portfolios 

♦ Peer production 

♦ Quality assessment 

♦ Initial and Continuous Vocational Education and Training (I -VET, C- 
VET), Continuous Professional Development (CPD), skills, competences 

♦ I ntroductory questions: 

a) Is quality in peer production processes with Web 2.0 applications 
manageable? 

b) Is quality manageable with traditional approaches (e.g. ISO, 
EFQM)? 

c) Is there a generic process model of peer production? 



B) Presentation of 3 QMPP Practice Cases 

Short presentation of 3 QMPP practice cases e.g. lAVANTE, www.azubi.net . 
Finish Elevator Company (optional) 



C) Discussion I : Case Assessment & Suggestions for I mprovement 

Three Speed Cafes on the three best ways to develop / to improve the 
quality of peer production in the 3 QMPP practice cases. 



D) Presentation of Changes from Web 1.0 / E-Learning 1.0 -> Web 



2.0 / E-Learning 2.0 



E-Learning 1.0 


E-Learning 2.0 






Quality assessed through 

experts 


Quality assessed through learners and 
peers 


Learning platform 


Personal Learning Environment 


Content 


User Created Content 


Curricula 


Learning diaries/ e-portfolios 


Course structure 


Communication 


Tutor availability 


1 nteraction 


Multimedia (Interactivity) 


Social networks / Communities of 
Practice (CoP) 


Acquisition processes 


Participation processes 
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E) Presentation of the Process Model of Peer Production 




« Creating 

0 (shared) authoring of courses, texts, resources, wikis; creating 
images, audio materials, video materials etc. 

« Validating 

0 validating content with subject matter experts, vaiidating content 
with peers, rating the content etc. 

« Editing 

0 shared editing, undertaking peer reviews, creating alternative 
navigational routes etc. 

« Enriching 

0 creating additional content materials, wikis, publishing individual 
works and team works, sharing or learning (b)logs, social 
bookmarking etc. 

« Updating 

0 - monitoring existing content, updating existing content, adding 

specific area content etc. 

« Benchmarking 

0 identifying of good cases and practices, identifying of good digital 
resources, sharing iearning experiences by sharing iearning 
(b)logs etc. 
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F) Presentation of the Peer Production Quaiity Matrix & reiated 
Research Questions 



O What is the object 


© What are the dimensions 


of the quality assessment? 


of the quality assessment? 


• Learning process 


• Pedagogical 


• Work/business process 


• Technological 


• Degree of 


• Economical 


communication/ participation 


• Institutional 


• Learning 


• Organisational 


achievement/outcomes 


• Cultural 


• Learning object, course 


etc. 


content, resource 




etc. 




© What are methods and 


O What are the stakeholders 


instruments 


of the quality assessment? 


to assess/develop quality? 


• Users 


• Self assessment 


• Experts 


• External assessment 


etc. 


• Peer review 




• Collaborative dialogue 




etc. 





G) Discussion II: Benchmarking the Quality Assessment in Peer 
Production Processes with Web 2.0 Applications 

Four Learning Cafes on how to benchmark the quality assessment of peer 
production processes with Web 2.0 applications by relating the Peer 
Production Quality Matrix with the Process Model of Peer Production 



H) Rapport from the Learning Cafes & Final Plenary Discussion 
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